[Python-Dev] Re: Some pattern annoyance

[Paul Moore]

On Wed, 2 Aug 2023 at 15:24, Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Partly because that's where the other discussants are (the network
> externality is undeniably powerful), and partly (I believe) because
> effective use of email is a skill that requires effort to acquire.
> Popular mail clients are designed to be popular, not to make that
> expertise easy to acquire and exercise.  Clunky use of email makes
> lists much less pleasant for everyone than they could be.
>
> I guess that's sad (I am, after all, a GNU Mailman developer), but
> it's reality.
>

Personally, I'm sad because some people whose contributions I enjoy (you
being one of them :-)) didn't move to Discourse. But like you say, it's how
things are.

Christian - you can make named constants using class attributes (or an
enum):

class A:
M = "M"

match seq:
case A.M, A.M, A.M, A.M, *r:
return 4*1000, r

Basically, the "names are treated as variables to assign to" rule doesn't
apply to attributes.

I'm not sure how helpful that is (it's not particularly *shorter*) but I
think the idea was that most uses of named constants in a match statement
would be enums or module attributes. And compromises had to be made.

Cheers,
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z4RJIDUDC475Y3A6UPXKDTGDTFJX2W5C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: A proposal to modify `None` so that it hashes to a constant

[Paul Moore]

On Sat, 3 Dec 2022 at 14:46, Yoni Lavi  wrote:

> > I think this is over-complicating things. I think the key merit of your
> > original proposal was its simplicity. Proposing more complicated ways of
> > getting the result you want is (IMO) unlikely to succeed, and is only
> > likely to cause people to become even more entrenched in their positions.
> > Can you give any explanation of why this proposal is better than your
> > original one, *apart* from "it's not been rejected yet"?
>
> It does sidestep the "There is nothing special about None" argument which
> IS the reason cited for rejecting the previous change.
>

OK, but it also reframes the problem as being about security, not about
reproducibility, which feels like trying to make the issue more "scary" in
the hope that people might care more.


> OTOH, you are still right, in the sense that I know that no matter what
> exactly I propose, there WILL be a way to view it negatively.


That's true of every proposal, and while I sympathise with your
disillusionment, I don't think looking at it that way will help your case.


> (BTW: did you see the 2nd flavor of my proposal in the P.S. where I
> *don't* get what I want)?
>

If by that you mean the "other way to implement the security fix", I saw it
but discounted it. I think the security aspect is a red herring. I
originally mentioned it as a "we should check" suggestion, but people who
know have all (as far as I can see) said there's no security risk here, so
I think we should drop the security related arguments.

I'm having trouble staying away from the discussion - I tried once and
> failed. Other people continue to talk about it, then I join again.
>

Me too :-) But I suggest seeing other people talking about it as a good
thing. Don't try to have the last word (something I find myself doing way
too often), but instead think that if people are talking, they are trying
to form their own views - and repeating yours won't add anything new, but
might distract people who were gradually coming to see your way of thinking.

Oscar Benjamin made a good argument recently in one of the threads. People
*are* helping make your point, you don't need to interject with "look, see,
this is what I meant!" every time. You've sown the seed, let it grow.


> But it might be best if I leave the discussion or at least my position as
> the person who's proposing this change.


Don't feel so tied to "owning" the proposal. You started a conversation,
it's got people thinking, wait and see what consensus forms. I still think
you pushed *way* too hard initially, and came across as not being willing
to listen to counter-arguments, but there's no reason you can't back off a
little and learn from that experience. Many proposals generate huge
discussion threads, but eventually fail because everyone digs their heels
in (this is what I think Chris Angelico perceives as "making proposals run
the gauntlet"). Letting ideas grow and develop, being willing to give
people time to decide, has a very different feel to it, but it requires the
proposer to let the debate happen, and *guide* it, rather than controlling
it. In my experience, not many new proposers know how to do this (or they
don't care enough to try).

This is a tiny change, almost insignificant in terms of benefits and
consequences. But it's a chance to learn the dynamics of making a proposal,
and understanding how to influence the direction of a discussion. Even if
the proposal ultimately fails, learning more about that is good. (Sorry,
re-reading that it sounds rather patronising. I don't mean it to, but I do
know that I see lots of people commenting that there's a lot more
"politics" and "people skills" to open source than they realised, and this
is a really good example of that in action, so I couldn't resist pointing
it out).


> Forget about the PR/issue and start over. Perhaps someone else needs to
> push for it. A core dev, so their opinion on the matter will count for
> something.


I really hope that the take-away here *isn't* that only the opinions of
core devs matter. It's true that only core devs can make changes, but if we
ever reach a point where community opinions aren't important, we've failed
badly. What *is* true, is that Python is big enough that *every* change has
to be considered in terms of the overall value to the whole community. Core
devs have to make those sorts of judgements a lot, so their views have
weight. With a community member, it's often not clear whether they are a
new contributor with a "great idea" but no experience in handling big open
source projects, or someone who's highly experienced in a different
community that we just don't know about. So "show that you can argue your
point and persuade people - or at least learn how to do so" is a challenge
non-core devs need to navigate to be heard. But it's not impossible to do
so, and it's not a failure if you struggle learning how to get your points
across.

Also, people are people, and we all 

[Python-Dev] Re: A proposal to modify `None` so that it hashes to a constant

[Paul Moore]

On Sat, 3 Dec 2022 at 10:57, Yoni Lavi  wrote:

> There's a number of Core devs that have taken strong positions against
> this change, citing various reasons ranging from "the addition of a
> function that returns a constant will cause bloat in the interpreter /
> needs to be tested / etc" to "what you really mean to ask for is set
> iteration stability, and we don't want that" to "identity based hashing is
> the default correct choice of a hashing function to use in any situation,
> unless we are forced by the requirements not to (even if it's
> disadvantageous compared to other choices)" to just straight appeals to
> authority ("rhettinger closed the issue on github so he must have done it
> for a good reason).
>

My position isn't an "appeal to authority", it's more a case of
acknowledging that core devs have a collective decision making role that is
normally exercised by consensus decisions with all but one or two of the
devs abstaining. In this case, so far only Raymond has not abstained, so
his view stands. Multiple core devs have contributed to these threads, and
likely more have read without commenting - but none of them have been
sufficiently convinced to state an opposing view on the tracker (which is
what matters here).


> I'm not sure if they actually believe what they say in all of these cases.
> To me, it sounds more like "please go away" than an honest argument on
> technical merit, but it matters little.
> I don't think anything can be changed with further technical discussion.
>

It's absolutely *not* "please go away" in my case. It's more "I see your
point, but I don't personally care enough to make an issue of the decision
on your behalf". Although I will admit that I have now reached the point
where I sort of wish this whole discussion *would* "go away" - whether by
you accepting that no-one's going to reverse the decision, or by another
core dev supporting the change on the tracker, rather than just posting
here and on Discourse. But I understand the temptation to just continue the
discussion without taking a formal stance (after all, I'm doing it myself).


> I do have another suggestion that I think merits a discussion. Maybe it
> will fare better. This change has a bit broader scope.
>

I think this is over-complicating things. I think the key merit of your
original proposal was its simplicity. Proposing more complicated ways of
getting the result you want is (IMO) unlikely to succeed, and is only
likely to cause people to become even more entrenched in their positions.
Can you give any explanation of why this proposal is better than your
original one, *apart* from "it's not been rejected yet"?

Seriously. No matter what your proposal, you need core dev support on the
tracker. IMO you stand more chance with your original proposal, in spite of
Raymond's rejection. But at this point, endlessly posting your views
everywhere isn't helping. Give people time to think and consider, and maybe
someone will decide to support the change. There's no urgency - 3.12 is a
year away, so that's the soonest this might be available.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G4OSEPLPHANUZUJEJA7YSN733INMDND3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Paul Moore]

On Mon, 17 Oct 2022 at 11:20, Denis Kotov  wrote:

> Stephen J. Turnbull wrote:
> > Denis Kotov writes:
> > > See:
> > > https://youtube.com/clip/UgkxyNe_dsZKinT_RT3UGb-BaP0SnvKteo2o
> > > No thanks, if it's not text I'm not going to look at it.  I don't have
> > time to watch videos with no explanation, and best guess is 90% of it
> > I already know, just as I don't think you've yet written anything I
> > didn't already know.
>
> It is just 10-20 seconds, I used YouTube Shorts to slice it :)
>

So while I agree with Stephen that if someone expects me to watch a video I
almost certainly won't (no matter how short) I watched this, just to make
the point. And indeed, it says nothing that I (and I imagine Stephen)
didn't know, and all it does is list a bunch of vulnerabilities and assert
that C++ can help with these. And it's presented by Herb Sutter, so yes,
I'm sure he knows what he's talking about. And indeed, I know C++ so I can
confirm that C++ has idioms and language features that you can use to help
with mitigating these vulnerabilities. But it's not a magic wand, you still
have to write the code to use these features. You can write pure C code in
C++, so there's no instant fix here, you have to **change the code** to
gain the benefits you suggest.

If you don't have a *specific* explanation of how Python can be improved by
rewriting it in C++, that takes into account all of the costs of such a
rewrite, then you won't get anywhere here. I could just as easily say "yay,
Rust is great, let's use Rust".

Or if you're that convinced, do the work. Make a fork of Python that uses
C++ instead of C, demonstrate that it's (in whatever way you want to
measure) "better" than the existing implementation, and be prepared to
defend that position. That's what multiple people have done, when
suggesting ways to speed up Python, ways to remove or mitigate the GIL,
etc. That route *works*, if your argument is sound. But just saying "you
should do X" and expecting people to just do the work for you is almost
certainly *not* going to work.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MIZGXZ3S6V52SPW3EH7Z76YCBDFZVJNM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Summary of Python tracker Issues

[Paul Moore]

On Fri, 13 May 2022 at 19:54, Brett Cannon  wrote:
>
> Can we shut this down or unsubscribe the weekly email?

My understanding was that it would be updated to get its information
from Github once the transition was complete. Is that not going to
happen now? I'm not particularly bothered, as I only really skimmed
the weekly email so it wouldn't be a great loss. But I agree, it
should either be modified or removed.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GU3S32Y23AO2WWQ2OMMBU37GYNGOBL4U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Starting a new thread

[Paul Moore]

On Tue, 10 May 2022 at 16:31, Barney Stratford
 wrote:
>
> Hello all.
>
> It occurs to me that creating threads in Python is more awkward than it needs 
> to be. Every time I want to start a new thread, I end up writing the same 
> thing over and over again:
>
> def target(*args, **kwds):
> ...
> t = threading.Thread(target = target, args = , kwargs= )
> t.start()
>
> This becomes especially repetitive when calling a target function that only 
> makes sense when run in a new thread, such as a timer.
>
> In my own code, I’ve taken to decorating functions that always run in new 
> threads. Then I can call the function using the usual function call syntax, 
> and have the new thread returned to me. With the decorator, the code reads 
> instead:
>
> @threaded
> def target(*args, **kwds):
> …
> t = target(*args, **kwds)
>
> This has a couple of advantages. I don’t have to import the threading module 
> all over my code. I can use the neater syntax of function calls. The 
> function’s definition makes it clear it’s returning a new thread since it’s 
> decorated. It gets the plumbing out of the way so I can concentrate more on 
> what my code does and less in how it does it.
>
> It feels like the right place for this decorator is the standard library, so 
> I’ve created PR #91878 for it. @rhettinger suggests that this is a bit 
> premature, and that we should discuss it here first. Thoughts?

I like this. I'd probably use it if it existed. Your example isn't
particularly compelling, though - calling the function "target" makes
sense when it's the target of the thread, but not so much when it's
the thread itself. Could you give a motivating example based on real
code, rather than just the "this is what the syntax would look like"
as you did above? Ideally showing what it would look like with and
without the decorator?

Things I *think* i get, but I'd like to see clearly in an example, include:

1. Does t = target(...) start the thread? I think it does.
2. Is it possible to create daemon threads? @threaded(daemon=True)
seems plausible. I suspect your PR doesn't have this, but it wouldn't
be difficult to add. But I don't actually know if it's worth adding,
TBH.
3. Can you create multiple threads for the same function? I assume t1,
t2, t3 = target(arg1), target(arg2), target(arg3) would work.

I'm probably being dense here - I'm pretty sure your proposal actually
is as simple as it looks (which is a good thing!) and I'm just
over-complicating things in my head.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FVETRDHASCD4ZYJ5YQG77LTG7R6TLIZO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Add -P command line option to not add sys.path[0]

[Paul Moore]

On Wed, 27 Apr 2022 at 16:50, Victor Stinner  wrote:
>
> Since I didn't get any serious review on my pull request since
> February, I created this thread on python-dev to get more people
> looking into this issue.

Pull requests don't get much visibility from the wider community - I
know I can't keep up with all the PRs submitted. Creating a thread on
python-dev seems reasonable. Doing so with about a week until feature
freeze seems less so. I'm not going to comment further on this
specific proposal, except maybe to respond to other people's comments.
I don't have enough time in the coming week to think through the
implications and possibilities myself.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HDHESOPY3LZPZGN5II67GZPNPMYZ4E7X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Add -P command line option to not add sys.path[0]

[Paul Moore]

On Wed, 27 Apr 2022 at 15:32, Victor Stinner  wrote:
>
> On Tue, Apr 26, 2022 at 11:46 AM Victor Stinner  wrote:
> > I propose adding a -P option to Python command line interface to "not
> > add sys.path[0]":
> > https://github.com/python/cpython/pull/31542
>
> My plan is to merge this change at 2022-05-05, the day before the
> Python 3.11 feature freeze,

Why leave it until the last minute? That just makes it harder to
revert if someone immediately finds a problem with it.

> unless someone has a good reason to not
> add this option or if you consider that we need more time to think
> about this issue.
>
> This issue is being discussed for 11 years, see for example:

It seems very rushed to propose this and implement it days before 3.11
freeze. If it's been an issue for 11 years, then (a) why didn't anyone
propose this solution months ago, and (b) surely it can wait another
year?

I don't have any particular objection to the feature, but whether you
mean it to or not, the short timescale gives the impression that
you're trying to rush something in without giving people time to
discuss or consider alternatives. People have other things to do, and
can't simply produce a response in a matter of hours. Apart from
anything else, aren't a lot of people who might be interested going to
be occupied with PyCon right now? Steve Dower mentioned some (IMO)
reasonable points, and you pretty much dismissed them without any
discussion. That doesn't seem like the right way to handle this. In
particular, I think the question of how this flag interacts with all
the other flags and settings that affect sys.path and how "isolated"
Python is, is an important thing to consider[^1].

Paul

[^1]: We've had multiple attempts to get locale and UTF8 handling
right, and now have a mess of flags, environment variables, and
options that frankly only the experts can understand. I fear that we
could end up with the same issue for "Python initialisation flags" if
we don't take a less rushed approach.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBQTRPWB5PJV4S3S2SFESYOQQ2B2E5CY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes

[Paul Moore]

On Sat, 23 Apr 2022 at 22:42, Rob Cliffe via Python-Dev
 wrote:
>
> UGH!
>
> I thought there was a general understanding that when typing was added
> to Python, there would be no impact, or at least minimal impact, on
> people who didn't use it.  (Raises hand.)
> Now we see an(other) instance of intention creep.

To be fair, none of this is needed unless you want to add type
annotations to your code. So it's still perfectly possible to ignore
all of this (which is what I am currently doing).

What I am concerned about is when users of libraries I write start to
claim that I "need" to add this sort of stuff to my code, so that they
can type check their code that uses mine, and/or they can get tooltips
for my APIs in their IDEs. That's where I think the biggest issue with
a proposal like this arises - the *social* pressure on people to adopt
typing, and all the complexities it adds. But again, that's not
something that's specific to this proposal, it's inherent in the whole
question of whether people add type annotations at all.

So I'm -1 on this proposal, but just because I fear I may be forced to
use it when I don't need to, rather than because I think it's a bad
idea per se.

Paul

PS To be open here, I do actually like type annotations in
straightforward situations - they have located some bugs in my code
for me, and tooltips in VS code *are* nice. What I don't like is not
being able to stick to simple stuff and not bother annotating the
complicated bits, or being pushed into over-strict annotations because
it's too hard (or verbose) to express dynamic, duck-typed constraints.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MOO3FUV3XKSKEGRIAOXSIVBMGPGFXYLV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Declarative imports

[Paul Moore]

On Fri, 8 Apr 2022 at 12:57, Daniel Pope  wrote:
>
> On Fri, 8 Apr 2022 at 12:23, Steve Dower  wrote:
> >
> > I've read the rest of the thread so far, and agree strongly that we
> > can't do this at the language/runtime level.
>
> You mean the hoisting, right?
>
> I don't see any reason why an import expression without hoisting would
> be impractical. But I'd like to hear your thoughts if you think it is.
>
> Desirability on the other hand is subjective. I think I actually do
> desire it, others are not bothered. I don't see strong arguments as to
> why you definitely wouldn't want it in your language.

OK, I'll be explicit. I don't want this in Python. Having imports at
the top of a file makes it much easier to see your dependencies. The
current language features, combined with community wide style guides
that discourage the use of local imports, make this straightforward,
while still providing means of doing local imports if needed.

This proposal isn't just about having a new syntax to do an on-demand
import. It's also about normalising the idea that people can pull code
from other modules without declaring the intent to do so up front.
While I don't dispute that in some circumstances (notably the REPL,
and throwaway scripts[1]) not having to add import statements would be
nice, I don't see how we'd limit usage to cases like that - and I
*really* don't want to have to work out where some weird inline import
happened while debugging a priority 1 bug in a 10,000 line code base
at 2am...

Also, it should be possible to do something like this using the
existing import machinery:

from magic_import import on_demand_loader as OD
...
# Many lines of code later
default_args = {
"start_date": OD.datetime.datetime(...)
}


Trivial proof of concept implementation:

class OD:
def __getattr__(self, name):
return __import__(name)

OD = OD()

print(OD.datetime.datetime.now())

Paul

[1] Of course, today's throwaway script usually turns out to be next
month's key component in a mission-critical software stack :-(
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HC4UJPBWJHU5JWIPQQKNJRMN2QBXHJVC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Declarative imports

[Paul Moore]

On Fri, 8 Apr 2022 at 10:27, Malthe  wrote:
>
> On Fri, 8 Apr 2022 at 08:51, Paul Moore  wrote:
> > Are you exaggerating for effect here or would this *actually* just expand to
> >
> > from datetime import datetime
> > default_args = {
> > "start_date": datetime(...)
> > }
>
> Yes – and of course that is just a snippet, an actual complete script
> will have lots of such imports.
>
> The point is that in a scripting situation (especially one in which
> you have many small scripts) the
> top-level import requirement for simple imports like `datetime`
> becomes rather verbose.

But Python's origin is in scripting situations, and explicit imports
for scripts has always been both the normal approach, and one of
Python's *strengths* ("Explicit is better than implicit"). Arguing
that explicit imports are a bad thing in (general) scripts is arguing
against decades of history and experience. If there's a reason why
*your specific context* would benefit from an abbreviated form, you
need to present it. But arguing that explicit imports are too verbose
for all cases of scripting isn't going to fly, frankly.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TMJ4ZI7J62C64PZXQU3WRLFA6FAKPRVB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Declarative imports

[Paul Moore]

On Fri, 8 Apr 2022 at 09:29, Malthe  wrote:

> A workflow definition which could be a 5-liner quickly
> becomes a 20-liner – consider for example:
>
> default_args = {
> "start_date": @datetime.datetime(...)
> }

Are you exaggerating for effect here or would this *actually* just expand to

from datetime import datetime
default_args = {
"start_date": datetime(...)
}

I'm not suggesting that there isn't a genuine benefit here, but your
example doesn't demonstrate what it is.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RRLFDL2BDF3UV646GVZVRXUSGM7X5FRQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Descriptions in unittest and avoiding confusion

[Paul Moore]

On Mon, 4 Apr 2022 at 17:22, Coyot Linden (Glenn Glazer)
 wrote:
> > I would welcome a multiline comment format that didn't involve docstrings.
>
> Err, sorry, I meant multiline string format.

I'm confused, what's wrong with """..."""? Triple quoted strings are
not exclusively for docstrings...
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OZ267ZAGWAMCC5WVMQX7TH2DYKNZBTVT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Paul Moore]

On Sat, 2 Apr 2022 at 02:30, Christopher Barker  wrote:
>
> On Fri, Apr 1, 2022 at 4:06 AM Steve Dower  wrote:
>>
>> The main difference is that 'immutables' offers you a stable/versioned
>> interface to use it, while the one that's in CPython is an internal
>> implementation detail. If one day we find a better design, we can just
>> switch to it, while 'immutables' probably can't. If we've exposed as a
>> public interface in the core runtime, it's much more complicated.
>
>
> I don't understand the issue here:
>
> If we expose a "frozendict" as built in python object then only the API of 
> that object needs to remain stable, not the implementation.
>
> And it seems that's an API that is already clearly defined.
>
> + 1 from me -- just the other day I was wishing it was there.

There would presumably need to be be a C API as well, and that would
probably expose more of the implementation unless handled carefully.
Without seeing an actual implementation, it's hard to know for sure.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y5RV73EOBXMQ252KWIZQYLUNCE3DW3OJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Wed, 30 Mar 2022 at 12:28, Steve Dower  wrote:
>
> On 30Mar2022 1132, Baptiste Carvello wrote:
> > Le 28/03/2022 à 18:44, Steve Dower a écrit :
> >> I think to most people "batteries included" doesn't necessarily mean
> >> "standard library," with all that implies. It just means "it came with
> >> the first thing that I installed" (or alternatively, "at no additional
> >> charge").
> >
> > A point I have not seen made, is that some uses really need *standard*
> > batteries.
>
> I've certainly not forgotten it, it's just every time I try to make the
> point it doesn't seem to help. So until I find a clear way to argue
> that, yes, a standard "datetime" value makes sense, but no, a standard
> "HTTP headers dictionary" is unnecessary, I'm avoiding bringing it up
> again ;)

I also strongly agree with this position, and do my best to argue it.
But as you say, it is hard to draw the line.

Here's some examples, which while not exhaustive, might give a flavour
of my views:

"Obviously" things that should be in the stdlib (basic data types or
algorithms):
re, datetime, collections, enum, dataclasses, decimal, fractions
(maybe borderline), statistics, itertools, functools, pathlib

Obvious stuff that's OS related:
os, io, threading, multiprocessing, concurrent.futures,
subprocess, asyncio, urllib.request

Core services:
sys, venv, sysconfig, warnings, atexit, gc, zipimport, runpy,
importlib, marshal, pickle

Standard development tools:
argparse, pdb, cprofile, typing (arguable in some ways)

Handling of key file and data formats:
struct, csv, json, tomllib, zlib, gzip, bz2 (borderline), zipfile,
tarfile, base64

Arguable cases of file formats, but with important use cases
xml (you may not like it, but it's still used a lot)
configparser (similarly, ini-style files are used a lot, for
better or worse)
email (this is an outlier, but the packaging metadata file format
uses it, so there's a bootstrapping issue if it's not in the stdlib)

I've missed a lot out, and I'm sure people will argue most of these,
but that's sort of the point - it's *hard* to come up with a blanket
statement on how we should approach "the stdlib", because there's so
much diversity in there. That's why my starting position is that we
should retain the stdlib - sure, we can remove (and add!) stuff, but
we should do so on a case by case basis, not based on some blanket
"minimise it because we don't have the manpower or we don't want to
support it" basis.

If writing and sharing small, one-off scripts that depended on
non-stdlib packages were *significantly* easier, I might revise my
position on some of the arguable cases. PEP 582 (Python local packages
directory) might make a big difference here. But at the moment, adding
a 3rd party dependency effectively alters what you're doing from
"writing a script I can share" to "developing an application" - and
that's a *big* step for many people. Quoting Nathaniel Smith from
https://mail.python.org/archives/list/distutils-...@python.org/message/YFJITQB37MZOPOFJJF3OAQOY4TOAFXYM/
(albeit in a slightly different context):

"""
This last point is important: as projects grow in complexity, and are
used by a wider audience, they often end up with fairly complex tool
specifications that have to be shared among a team.
"""

My argument is that relying only on the stdlib allows people putting
off the need to take that step up in complexity, and we shouldn't
dismiss how significant that is.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LYKBB6YVZYQTJHPVRR5YMWE5D6YNSENY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Tue, 29 Mar 2022 at 00:37, Toshio Kuratomi  wrote:

> One thing about talking about "make urllib more like requests" that is 
> different than any of the other libs, though, is that requests aims to be 
> easier to use than anything else (which I note Chris Barker called out as why 
> he wanted urllib to be more like it).  I think that's important to think 
> about because i think ease of use is also the number one thing that the 
> audience you talk about is also looking for.

Agreed, but one thing that I would note is that the more people ask
for changes of this sort of scope, the more this fuels the argument
that maintaining the stdlib is a lot of work. It's fairly obvious that
none of the core developers have the time or energy to invest in the
sort of urllib API change you're suggesting here, so realistically
this simply gives the impression "people don't want urllib, they want
requests". The net result is not that urllib becomes more like
requests, but rather that there's extra pressure to *drop* urllib, and
point people to requests.

Personally, I want *both* urllib and requests. I want urllib for when
I don't want a 3rd party dependency, and requests (or httpx, or
urllib3, ...) for when I can afford one. And if the cost of having
that is that urllib has a less user-friendly API, then I'm willing to
live with that.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/77D3WHKPB2FS2JKAC3EPTPDH6UDQIAFD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Mon, 28 Mar 2022 at 20:37, Steve Dower  wrote:

> Please let us know
> *publicly* if you want to become the maintainer for a stdlib module and
> then we can support them, but if nobody is willing/able/ready to care
> for them it's irresponsible for us to keep shipping them to users.

I'm sorry for picking on a single point you made and continuing to
simply state my contrary view, but I think it's important.

IMO, it's irresponsible of us to *remove* functionality if there's no
suitable alternative (and remember, my definition of "suitable"
includes "appropriate for cases where a PyPI library isn't acceptable
for some reason"). Of course, it's not OK for us to promise a level of
support that we can't or won't provide, but *we don't provide such
promises*. We say that we support the stdlib, but there's no actual
statement of what that means - so it's not clear to me how we're
failing to deliver "enough". Nor is it at all clear to me that dumping
the responsibility on an external maintainer will be any better - most
likely it just lets *us* feel better about the situation while not
actually being better for the end user.

If we were to define some sort of actual support guarantees, we might
be having a different conversation. But I'm unclear how a volunteer
organisation can provide meaningful promises that we wouldn't be able
to deliver for the stdlib as it stands now.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OSYFNLIULB56EVSG57QXKIW4NRWI4CNZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Mon, 28 Mar 2022 at 17:45, Steve Dower  wrote:
>
> I think to most people "batteries included" doesn't necessarily mean
> "standard library," with all that implies. It just means "it came with
> the first thing that I installed" (or alternatively, "at no additional
> charge").

I think that for users it means "came as part of the official
installer" where "official" means "whoever I chose to trust" - which
in turn often means "python.org".

For package maintainers it means "the bunch of stuff I can assume
exists when someone says they have python" - which is why distros that
debundle parts of the stdlib get so much grief. Also note that "just
add a dependency" is not a simple request for a library maintainer -
if I depend on requests, and someone using my library uses httpx, then
I'm imposing a potential transitive dependency conflict on them. So
using stdlib functionality is a non-trivial advantage for library
maintainers (of course, sometimes the benefits of using a 3rd party
library outweigh this, but not always).

> Given there are *plenty* of existing distros out there that install more
> than just the standard library for the benefit of their users, and yet
> the people asking for a fatter stdlib don't seem to want them, perhaps
> we just need to officially endorse one?

But people don't use those existing options presumably for a reason.
Having the core devs say "distro X is approved" won't remove those
reasons. And removing the python.org distro because X is now approved
will create a problem for those users.

And none of this alters the fact that "stdlib" means "can be assumed
to be present in all Python installations" for some use cases.

> I've got no issue preinstalling a curated sumo.txt into the Windows
> installers, provided we're clear about support (i.e. none from us) and
> someone besides me is tracking everything in that curated list. That
> probably requires new volunteers who are somehow already trustworthy,
> but that should be easier than people to actually write and maintain new
> functionality on stdlib timelines.

"Support" is a somewhat ambiguous matter in the context of
volunteer-supported software. The point of having a stdlib is that
people know where to report issues, not that we're necessarily any
more responsive than a 3rd party maintainer. There's a value in
knowing that if you find an issue with "anything you installed as part
of the basic Python install", you report it in the same place. And
having the documentation all in one place. Etc.

Yes, people come along and make a big fuss about how they expect more
in the way of support ("this bug is urgent, my production system is
failing because of it") and yes, people with that attitude have a
negative effect on core developer morale. But we don't promise fixes,
or solutions. At best there's an implication that the core devs care
about the stdlib in some sort of vague, generalised sense. (I care
about the health of the stdlib, but I'm not likely to try to fix a bug
in ossaudiodev, for example).

If we did try to mandate something more concrete, then maybe the
conversation would be different. But I'm not aware of anyone proposing
anything like that.

> We would still have to have the "light" distribution, as many of our
> users wouldn't be able to use a "this is full of unsupported stuff"
> package, but at least most users could be pointed to the bigger install
> by default and not even care that the batteries just came in the same
> package and aren't actually an integral part of the core product.

To be honest, I feel like I'm just reiterating stuff I've said before
here, and I think the same is true of the points I'm responding to. Is
there actually any new development here, or is it just a repeat of the
same positions people have expressed in the past, based on the same
underlying realities? (I'm not *against* going over the debate again,
it helps make sure people haven't changed their minds, but it's
important to be clear that none of the practical facts have changed,
if that is the case).

I should also point out (as someone heavily involved in Python's
packaging ecosystem) that I don't believe that the existing packaging
infrastructure is sufficiently robust to support a world where the
Python stdlib was all delivered as external packages. And even if it
were, bootstrapping something like pip would be a nightmare - for a
start, we don't vendor libraries that include C extensions, so we
could only handle pure Python stdlib modules - and even vendoring all
of them (and all of our vendored tools' stdlib dependencies) would be
a nightmare.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SKIKUC4U3L74G43BNC5SDCXFYUY3G447/
Code of Conduct: 

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Sun, 27 Mar 2022 at 17:11, Christopher Barker  wrote:
>
> With the json package included, all they need to do is `import json`. If that 
> wasn't there, they's look in PyPi for a JSON implementation, and find an 
> absolutely astonishing number of options. I just did a search for "JSON"
>  on PYPI, and it's HUGE -- most of them are for specialized JSON-using 
> protocols of some sort. I was actually really surprised that couple I know 
> about of the top of my head (ujson, orjson) are actually hard to find.
>
> "You can just pip install it" is really not a good solution.
>
> In fact, this is an example, I think, of where we should put some effort into 
> making the included batteries better -- it's great to have a JSON lib built 
> in, but it's too bad that it's not best-of-bread by pretty much any 
> definition (speed, memory performance, flexibility) -- there are quite a few 
> feature requests open for it -- it would be nice to actually implement some 
> of those. (but yes, that's a lot of work that someone(s) would have to do)
>
> Back to the topic at hand, rather than remove urllib, maybe it could be made 
> better -- an as-easy-to-use-as-requests package in the stdlib would be really 
> great.

I think that's where the mistake happens, though. Someone who needs
"best of breed" is motivated (and likely knowledgeable enough) to make
informed decisions about what's on PyPI. But someone who just wants to
get the job done probably doesn't - and that's the audience for the
stdlib. A stdlib module needs to be a good, reliable set of basic
functionality that non-experts can use successfully. There can be
better libraries on PyPI, but that doesn't mean the stdlib module is
unnecessary, nor does it mean that the stdlib has to match the PyPI
library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it
can be, and not demand that it turn into requests. Requests is there
if people need/want it (as is httpx, and urllib3, and aiohttp). But
urllib is for people who want to get a file from the web, and *not*
have to deal with dependencies, 3rd party libraries, etc.

The "batteries included" standard library and PyPI complement each
other. Neither is redundant, and neither implies the other is
unnecessary.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E2FGXTYOEYLC6GSJGMWBQKHOO62LZPHQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

[Paul Moore]

On Sun, 27 Mar 2022 at 00:52, Ethan Furman  wrote:
>
> [apologies for the late post, just found this in my drafts folder]
>
> On 2/7/22 12:49 AM, Stéfane Fermigier wrote:
>
> > 3. Overall, I think the days where "battery included" was a positive 
> > argument are over
>
> I strongly disagree.  Being able to download something and immediately get 
> something to work and see results is hugely
> rewarding; on the other hand, having to research, find, compare & contrast 
> available third-party modules (especially for
> new-comers) can be extremely discouraging.

100% agreed.

I think people massively underestimate how frustrating it can be to
have to hunt out some piece of functionality that *doesn't* come with
the base download. Or worse, to find 5 different implementations, and
have no idea which one you "should" use.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2BIMFGNREXPOAFKGNVC7FZMT5LNVNVAY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Paul Moore]

On Sun, 6 Feb 2022 at 16:51, Christian Heimes  wrote:

> The urllib package -- and to some degree also the http package -- are
> constant source of security bugs. The code is old and the parsers for
> HTTP and URLs don't handle edge cases well. Python core lacks a true
> maintainer of the code. To be honest, we have to admit defeat and be up
> front that urllib is not up to the task for this decade. It was designed
> written during a more friendly, less scary time on the internet.
>
> If I had the power and time, then I would replace urllib with a simpler,
> reduced HTTP client that uses platform's HTTP library under the hood
> (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten,
> maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or
> aiohttp are much better suited than urllib.
>
> The second best option is to reduce the feature set of urllib to core
> HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter,
> more standard conform parsers for urls, query strings, and RFC 2822
> instead of RFC 822 for headers.

I'd likely be fine with either of these two options. I'm not worried
about supporting "advanced" uses. But having no way of getting a file
from the internet without relying on 3rd party packages seems like a
huge gap in functionality for a modern language. And having to use a
3rd party library to parse URLs will simply push more people to use
home-grown regexes rather than something safe and correct. Remember
that a lot of Python users are not professional software developers,
but scientists, data analysts, and occasional users, for whom the
existence of something in the stdlib is the *only* reason they have
any idea that URLs need specialised parsing in the first place.

And while we all like to say 3rd party modules are great, the reality
is that they provide a genuine problem for many of these
non-specialist users - and I say that as a packaging specialist and
pip maintainer. The packaging ecosystem is *not* newcomer-friendly in
the way that core Python is, much as we're trying to improve that
situation.

I've said it previously, but I'll reiterate - IMO this *must* have a
PEP, and that PEP must be clear that the intention is to *remove*
urllib, not simply to "deprecate and then think about it". That could
be making it part of PEP 594, or a separate PEP, but one way or
another it needs a PEP.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KT6TGUBTLLETHES2OVVGZWSGYC5JCEKC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Paul Moore]

On Sun, 6 Feb 2022 at 14:15, Victor Stinner  wrote:
> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.

Also, I'm -1 on deprecating as a way of saying we *might* remove the
module, but haven't decided yet. That isn't (IMO) what deprecation is
for, and it doesn't give users a clear message, as maybe they'll be
fine continuing to rely on urllib. The net result would likely to be
for people to simply become more inclined to ignore deprecation
warnings.

Conversely, if the idea is to deprecate, and then in a couple of years
say "well, it's been deprecated for a while now, so let's remove it"
then that seems to me to be a rather cynical way of deflecting
arguments, as we can say now "well, it's only deprecation", in spite
of the fact that the real intention is to remove.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ICHMNBE7PMOHCGXLT4REP2HJZAGSOCHJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: It's now time to deprecate the stdlib urllib module

[Paul Moore]

Strong -1 from me.

urllib.request may not be "best practice", but it's still extremely
useful for simple situations, and urllib.parse is useful for basic
handling of URLs.Yes, the more complex aspects of urllib are better
handled by external packages, but that's not sufficient argument for
removing the package altogether. There are many situations where
external dependencies are unsuitable. Also, there's quite a lot of
usage of urllib in the stdlib itself - how would you propose to
replace that?

In addition, pip relies pretty heavily on urllib (parse and request),
and pip has a bootstrapping issue, so using 3rd party libraries is
non-trivial. Also, of pip's existing vendored dependencies,
webencodings, urllib3, requests, pkg_resources, packaging, html5lib,
distlib and cachecontrol all import urllib. So this would be *hugely*
disruptive to the whole packaging ecosystem (which is under-resourced
at the best of times, so this would put a lot of strain on us).

In any case, why is this being proposed as a simple posting on
python-dev? There's already PEP 594 for removals from the stdlib. If
you have a case for removing urllib, I suggest you get it added to PEP
594, so it can be discussed and agreed properly, along with the other
removals (none of which is remotely as controversial as urllib, so
there's absolutely no doubt in my mind that this would need a PEP
however it was proposed).

Paul

On Sun, 6 Feb 2022 at 14:15, Victor Stinner  wrote:
>
> Hi,
>
> I propose to deprecate the urllib module in Python 3.11. It would emit
> a DeprecationWarning which warn users, so users should consider better
> alternatives like urllib3 or httpx: well known modules, better
> maintained, more secure, support HTTP/2 (httpx), etc.
>
> I don't propose to schedule its removal. Let's discuss the removal in
> 1 or 2 years.
>
> --
>
> urllib has many abstraction to support a wide range of protocols with
> "handlers": HTTP, HTTPS, FTP, "local file", proxy, HTTP
> authentication, HTTP Cookie, etc. A simple HTTP request using Basic
> Authentication requires 10-20 lines of code, whereas it should be a
> single line.
>
> Users (me included) don't like urllib API which was too complicated
> for common tasks.
>
> --
>
> Unhappy users created multiple better alternatives to the stdlib urllib 
> module.
>
> In 2008, the "urllib3" module was created to provide an API designed
> to be as simple as possible for the most common HTTP and HTTPS
> requests. Example:
>
>req = http.request('GET', 'http://httpbin.org/robots.txt').
>
> In 2011, the "requests" module based on urllib3 was created.
>
> In 2013, the "aiohttp" module based on asyncio was created.
>
> In 2015, new "httpx" module was created:
>
> req = httpx.get('https://www.example.org/')
>
> Not only httpx has a regular "synchronous" API (blocking function
> calls), but it also has an asynchronous API!
>
> Sadly, while HTTP/3 is being developed, it seems like in this list,
> httpx is the only HTTP client library supporting HTTP/2 currently :-(
>
> For HTTP/2, I also found the "httplib2" module.
>
> For HTTP/3, I found the "http3" and "aioquic" modules.
>
> --
>
> Let's come back to urllib:
>
> * It's API is too complicated
> * It doesn't support HTTP/2 nor HTTP/3
> * It's barely maintained: there are 121 open issues including 3 security 
> issues!
>
> The 3 open security issues:
>
> * bpo-33661 open 2018;
> * bpo-36338 open in 2019;
> * bpo-45795 open in 2021.
>
> Usually, it's bad when you refer to an open security issue by its
> creation year :-(
>
> The urllib module has long history of security vulnerabilities. List
> of *fixed* vulnerabilities:
>
> * 2011 (bpo-11662):
> https://python-security.readthedocs.io/vuln/urllib-redirect.html
> * 2017 (bpo-30119):
> https://python-security.readthedocs.io/vuln/urllib-ftp-stream-injection.html
> * 2017 (bpo-30500):
> https://python-security.readthedocs.io/vuln/urllib-connects-wrong-host.html
> * 2019 (bpo-35907):
> https://python-security.readthedocs.io/vuln/urllib-local-file-scheme.html
> * 2019 (bpo-38826):
> https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex.html
> * 2021 (bpo-42967):
> https://python-security.readthedocs.io/vuln/urllib-query-string-semicolon-separator.html
> * 2021 (bpo-43075):
> https://python-security.readthedocs.io/vuln/urllib-basic-auth-regex2.html
> * 2021 (bpo-44022):
> https://python-security.readthedocs.io/vuln/urllib-100-continue-loop.html
>
> urllib is a package made of 4 parts:
>
> * urllib.request for opening and reading URLs
> * urllib.error containing the exceptions raised by urllib.request
> * urllib.parse for parsing URLs
> * urllib.robotparser for parsing robots.txt files
>
> I propose to deprecate all of them. Maybe the deprecation can be
> different for each sub-module?
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe 

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Paul Moore]

On Mon, 17 Jan 2022 at 06:52, Denis Kotov  wrote:
> > And that's why you need to do more work than arguing that in principle
> > C++ is just a better language than C.  We've been hearing that for 4
> > decades now (at least we greybeards have), and we've discovered that
> > for many existing applications, C++ may be better but the cost of
> > converting large swaths of C code to equivalent C++ that passes all
> > tests is too big.  Python may very well be one of them.
> > So if you're not going to do the work to demonstrate big wins from
> > using C++ instead of C in actual Python implementation code, I think
> > you're wasting your posts.
>
> I thought about it, but will CPython accept the PR for this changes if it 
> would show benefits ?

We can't say that, no. The point here is that if you *don't* write
such a PR and demonstration, nothing will change. If you do, then
*maybe* it will get accepted, it depends if the benefits demonstrated
by the PR are sufficient.

The question is whether just having a *chance* of getting it in is
sufficient to persuade you to produce such a PR.

Paul.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QNUXQ66OKNET5AF6PNMZ5T6ZJWUTU7HV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Changing PySequence and PyMapping checks

[Paul Moore]

Personally, I dislike the presumption that you state in the bpo, that
typing and linters have changed things so that being a sequence is more
closely tied to implementing the Sequence abc these days. I consider it a
flaw in typing and linters if that’s the case, not something to embrace.

Paul.

PS Writing on mobile, sorry for any typos

On Sat, 15 Jan 2022 at 04:26, Bar Harel  wrote:

> There's a long time issue of trying to differentiate mappings and
> sequences in the C-API in a fast and reliable way.
>
> Due to recent changes, we might be able to do so at last, by checking
> tp_flags + str/bytes/bytearray which are considered unique.
>
> This might be a breaking change in the Stable ABI promise but one that can
> be considered a bug fix as the current behavior is lacking.
>
> What do you think?
>
> bpo: https://bugs.python.org/issue46376
>
> Best regards,
> Bar Harel
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WTIKR5DGMZRKRRAZGFTD4EPG5ITFHOVS/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T4LYOK4BQVM4VPMBYKPQP6LYT3CAGNIC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 677 (Callable Type Syntax): Final Call for Comments

[Paul Moore]

On Thu, 13 Jan 2022 at 08:48, Nick Coghlan  wrote:
>
> The existing section on runtime behaviour in the PEP doesn't look ready for 
> SC consideration, since it essentially says "this hasn't been defined yet".

+1. The runtime behaviour needs to be specified. Otherwise this PEP is
a spec for how type checkers work, but it fails to say how CPython
itself will work :-(

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DR5WITUYO4MA5GADVQLQFUFDTKY26LN4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Suggestion: a little language for type definitions

[Paul Moore]

On Mon, 10 Jan 2022 at 23:04, Greg Ewing  wrote:
>
> Most of the effort of learning to read type expressions, in any
> language, is learning their semantics, which is very different
> from value expressions. Whether they happen to superficially
> resemble other parts of the language is pretty much irrelevant
> to me.

While this is true, what *is* a problem with having a different
language, is that you have two very similar syntactic constructs
("expression" and "type expression") and you have to learn which can
be used where. And that's actually not as trivial as you'd think at
first.

The typing.cast function takes a type expression as its first
parameter. If type expressions are not expressions, syntactically,
then that means that typing.cast(MyType, obj) is *not* syntactically a
function call. So what is it? Say it's a new syntactic element, a
"cast expression". Then the syntax for an expression must be extended
to include cast expressions. And what *is* the syntax of a cast
expression? "cast_literal(type_expr, expr)"? Then what's
"cast_literal"? The value "typing.cast"? What about "from typing
import cast", or "import typing as t" with "t.cast", or "from typing
import cast as typecast", or even "my_cast = getattr(typing, "cast")?

Suddenly there are a huge number of unanswered questions to be
considered, and the answers to those questions need to be learned by
people.

Paul.

PS Having said that, existing type checkers may or may not recognise
all of the possible ways to call the cast function that I described.
Do we want to mandate what type checkers do as part of the language
definition? Do we want to restrict the language definition based on
what 3rd party type checkers can achieve?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LVLWHVSIF3WKDHFPF5MSOYEFR37FBXW2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 679 – Allow parentheses in assert statements

[Paul Moore]

On Mon, 10 Jan 2022 at 11:43, Batuhan Taskaya  wrote:
>
> > Does someone know if linters like pylint or pylakes current warn on 
> > "assert(test, msg)" statements?
>
> I believe so, both the python itself (through SyntaxWarnings) and
> other linters warn about this.

❯ py
Python 3.10.1 (tags/v3.10.1:2cd268a, Dec  6 2021, 19:10:37) [MSC
v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> assert (1==0, "Hello")
:1: SyntaxWarning: assertion is always true, perhaps remove parentheses?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SNTCRCIJCMLK7AYCQIIZ2XTRN5OX6J7M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Paul Moore]

To me, these are mostly convincing examples that people need to name
parts of a complex type like this :-)

I don't actually find any of the syntaxes better than any other. They
are all a bit bad, but I view that as the fault of the complex nested
types, not the syntax (hence my preference for naming things). It's
hard to give examples of how I'd name things, because you need to know
the "business logic" to name things well, and I don't for these.

Frankly, if people were writing normal Python expressions like this,
everyone would be telling them that Python isn't about writing
one-liners and they should factor out named subexpressions. Why should
type annotations be any different?

Paul

PS If I wanted any improvement for callables, it would be to use
parentheses for the arguments rather than square brackets (which look
too "heavy" in my view). So Callable[(int, str), bool] rather than
Callable[[int, str], bool]. I understand why the outer brackets have
to be square (abuse of indexing notation) but as far as I can see
there's no reason the inner ones need to be (OK, for single arguments,
Callable[(int,), bool] is a bit clumsy...)

On Tue, 21 Dec 2021 at 22:10, Steven Troxler  wrote:
>
> In the example I was aiming for something easy to understand that produced a 
> type illustrating potential problems of PEP 677, which is at its worst when 
> there are callables in both argument and return position. I don't have a 
> great real-world example of this worst-case, most of what I've seen involves 
> simpler and the current PEP 677 proposal isn't as bad.
>
> As for formatting I agree that I wouldn't hand-write the type as
> ```
> ((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool]
> ```
> but a lot of code formatters might fight me on this, so I think it's worthy 
> of consideration, and I was trying to illustrate a readability problem and 
> possible fix. Formatting the code nicely so that my proposal looks good would 
> have meant not really engaging with the concern.
>
> If you're looking for examples from real code where Callable is unweildy, 
> Pradeep collected a few at [1] from typeshed although most of them look just 
> fine with the current PEP 677 proposal. A couple examples:
>
> Callable[[AnyStr, Callable[[AnyStr, AnyStr, AnyStr], AnyStr]], AnyStr]
> Callable[[Optional[str], tuple[_Marshallable, ...]], Union[Fault, 
> tuple[_Marshallable, ...]]]
> Callable[[str, Callable[[Iterable[str], str], str]], None]
>
> versus the same types written using the current PEP 677 syntax:
>
> (AnyStr, (AnyStr, AnyStr, AnyStr) -> AnyStr) -> AnyStr
> (Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, 
> tuple[_Marshallable, ...]]
> (str, (Iterable[str], str) -> str) -> None
>
> versus with outer parentheses:
>
> (AnyStr, (AnyStr, AnyStr, AnyStr -> AnyStr) -> AnyStr)
> (Optional[str], tuple[_Marshallable, ...] -> Union[Fault, 
> tuple[_Marshallable, ...]])
> (str, (Iterable[str], str -> str) -> None)
>
> and another idea, requiring both outer parentheses and argument parentheses:
>
> ((AnyStr, ((AnyStr, AnyStr, AnyStr )-> AnyStr)) -> AnyStr)
> ((Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, 
> tuple[_Marshallable, ...]])
> (str, (Iterable[str], str) -> str) -> None)
>
> To me, these are convincing examples of where Callable is hard to read and an 
> arrow syntax is easier, but that doesn't necessarily mean it's worth the 
> price of new syntax.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/KWQVEQDZFBZDQDLOGATXSFN7KSF4WVYJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CMDXHQPLJMGCJLL2SVPFJLC3CRE5AUYU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Syntax PEP

[Paul Moore]

On Tue, 21 Dec 2021 at 18:35, Steven Troxler  wrote:
>
> I've been thinking about readability hard because I share many of your 
> concerns about readability.

Before I comment on syntax, I'd like to question the example:

> An example
> =
>
> The function
> 
>
> To get at what I mean, here's a nice simple function:
> ```
> def zip(f, g):
> def zipped(x: int, y: str):
> return f(x), g(y)
> return zipped
> ```

So my first question is why a generic function like this would limit
the parameters x and y to int and str. Surely this is an entirely
workable function for *any* arguments with the appropriate "shape"? So
how is this a good example of somewhere you'd use types like
Callable[[int], bool] (by the way, where did that "bool" appear from?)
in a signature?

My immediate thought when seeing that zip function is that its type is
"obvious"¹, (takes two functions, returns a function which takes 2
arguments matching the args of the 2 functions and returns a tuple
matching the results of the 2 functions). That type is almost
impossible to express clearly, because it needs pretty complex
generics. But the more important point is that I'd never, **ever**,
want to write that type out longhand. I'd expect a type system to
either infer the type, and it would be anonymous¹, or to give up and
treat it as untyped. One thing I absolutely would not want to do is
over-constrain any of the arguments just to make it possible to write
the type down.

The real issue with this function, in my view, is not expressing
callables, but rather generic type variables ("return a function whose
first argument has the same type as the single argument of the
function which is the first argument of this function..." !!!)

Can you suggest a more "real world" function as an example, which
focuses on the callable syntax and doesn't use things like arguments
called f and g? Maybe a GUI callback with a function argument like
on_click?

¹ By which I mean intuitive, not easy to express in words!!! :-)
² Some languages may allow syntax like `typeof(zip)` to refer to that
anonymous type, but that's a separate point.

> Here's it's type using typing.Callable:
> ```
> typing.Callable[
> [typing.Callable[[int], bool], typing.Callable[[str], float]],
> typing.Callable[[int, str], tuple[bool, float]
> ]
> ```
> which seems ugly. It's actually not bad compared to a lot of production code, 
> but this is the kind of thing that led to PEP 677.

It's massively over-constrained. I assume that's because you're trying
to make a point about Callable[] rather than about generics, but can
you give a realistic example that *doesn't* involve over-constraining
higher order functions?

On a side note, why not name at least some of those function types?
And why not use "from typing import Callable"? It feels like you're
not making enough effort to make your example readable, which
undermines your point as a result.

> ((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool]

To your credit, you've made this pretty unreadable, which gives some
balance here :-) Seriously, making it a one-liner with all those ->
arrows is a disaster. Changing the location of the parentheses doesn't
alter that at all.

Rewriting as a multi-line expression:

(
(int) -> float,
(str) -> bool
) -> (int, str) -> tuple[float, bool]

helps quite a bit, but returning a function looks bad here. We're not
writing Haskell, you know ;-)

I'd prefer a "mixed" notation here:

(
(int) -> float,
(str) -> bool
) -> Callable[(int, str), tuple[float, bool]]

I don't honestly think there's a readable "function returning a
function" form here - the chained -> tokens is just awkward. Although
that's clearly a matter of preference, there's never going to be an
objective answer here.

> Here’s the type if we change the syntax to put the right parenthesis after 
> the return type:
> ```
> ((int -> float), (str -> bool) -> (int, str -> tuple[float, bool])
> ```
>
> To my eyes, most of the pain points are now eliminated.
> - there’s never a double-arrow due to callable in return position
> - even for argument types, to my eyes it’s now easier to read

To my mind, the eye is still drawn to the arrows, and the readability
is no better. And the parentheses give me a lisp vibe, for reasons I
can't really pin down but which makes this version *less* readable.

> An added bonus is we no longer have to think about double-arrows when a 
> callable type is in the return position of a function, e.g.:
> ```
> def f() -> (int, str -> bool): ...
> ```

Still looks like double arrows to me, I'm afraid. The parentheses
don't group strongly enough to override the "chain of arrows"
impression.

> And it solves another major usability problem we found - writing optional 
> callables - because it’s now no problem at all to write
> ```
> (int, str -> bool) | None
> ```
> as the type of an optional callable argument.

I guess, but it feels like punctuation 

[Python-Dev] Re: RFC on Callable Syntax PEP

[Paul Moore]

I agree. The same concerns and reservations apply for me.

On Sat, 18 Dec 2021 at 21:13, Terry Reedy  wrote:
>
> Batuhan expresses my concerns better than I could, so I just add my
> agreement.
>
> On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
>
> > tl;dr: I find it very troubling that we are going on a path where need
> > to increase the language complexity (syntax) only in the cause
> > 'easier' typing. So I am opposed to this change.
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/G6PD5KYVI3CCAPFEOQ3VT7QJAN3Y6Z6W/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LS7DT7QVJ5RO5VX4IVCBMVQILGD2UQMQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Paul Moore]

On Wed, 8 Dec 2021 at 12:42, Petr Viktorin  wrote:
> > I assume it would be insensitive to ask whether we could just get rid of
> > the stable ABI altogether and focus on the limited API? Just tell
> > everyone they have to rebuild binary wheels for every Python feature
> > release. Presumably the deprecation of the stable ABI itself would
> > require a two-release waiting period. But maybe it would be worth it,
> > given how subtle it is to do the historical research about even a single
> > function.
>
> A honest question wouldn't be insensitive. Thanks for asking!
>
> The part where you don't need to rebuild extensions (not just wheels) is
> the main reason for both Stable ABI and the Limited API.
> Without it, there might be some reduced API to focus on, but it wouldn't
> be this feature.

This I think is the critical point here. I don't follow the API/ABI
debate closely, but I do know that the number varieties of C extension
builds some projects have to make is a serious overhead. Being able to
target the stable ABI and say "this will work for all (essentially)
versions of CPython" is a huge benefit, as it reduces the number (and
hence the burden) significantly.

Not having to amend your source for each CPython release feels to me
like a subset of this, and loses a major advantage, so my personal
view is that the stable ABI is far more beneficial than the limited
API (the limited API is what you fall back to if you're unable to use
the stable ABI).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KCAIYXMKY7RZULJNG7BGM5OQ7RF5Z266/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Fri, 3 Dec 2021 at 00:10, Shantanu Jain  wrote:
>
> @Paul
>
> > ... missing resource is a central set of typing documentation that includes 
> > examples, FAQs and best practices as well as reference materials
>
> Like Sebastian, I agree, and this is something we're making progress on.

That's great to know - it's easy for me to say "we need more docs" but
I know it's hard to produce them. I'm glad it's on the radar :-)

> > ... easy way of testing that the stubs are correct
>
> mypy ships with a tool called stubtest. It's what typeshed uses to validate 
> itself against CPython. I recently wrote up documentation for it, see here: 
> https://mypy.readthedocs.io/en/latest/stubtest.html

Thanks, I wasn't aware of that. Although TBH I like the suggestion of
using typeshed as a kind of staging point for interested users to
develop annotations, which can then be moved back into the library
once they've been proven stable.

I'm still surprised that mypy doesn't check stub files though. If I
write stubs for my external API, and then later decide I want to add
types to my internal functions, to do static checking on my library,
am I therefore required to move the annotations out of the stub file
into the main code? What if I want to avoid the runtime overhead of
importing the typing module? Or I want to keep the source code clear,
by not making function definitions into multi-line monsters because of
type declarations?

> > A lot of the frustration I see being expressed here (including my own) 
> > seems to come from the fact that it's so difficult to actually take that 
> > sort of "I can ignore it if I don't use it"
>
> Thanks for not staying quiet and helping us make typing better. In 
> particular, I think one takeaway from this thread is that a lot of our 
> documentation is focussed on "how to get started with typing my code", and 
> not much addressed at library authors with limited time to deal with typing 
> questions (like "what is a PEP 561" and "if my type hints are incomplete does 
> that cause problems for my users")

That's a good point that I hadn't spotted - thanks for picking up on it!
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CRBIOVH4K5VFIIPB2D6EJCIVWJXGITEU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Wed, 1 Dec 2021 at 12:08, Sebastian Rittau  wrote:

> Please note that users of you library usually won't care that the library 
> uses type hints. It's more important that there are type hints for the API, 
> which can also be supplied using a stub file.

I tried that route, but I was informed that if I do that, mypy will
not check my stubs against the source code. Which means that there's
no easy way of testing that the stubs are correct - is that accurate?

Paul

PS I appreciate the links you posted to various typing forums, but IMO
the most critical missing resource is a central set of typing
documentation that includes examples, FAQs and best practices as well
as reference materials. Typically when I hit an issue with types, I
want to research my own solution, not ask a question, which requires
me to bother other people, as well as interrupting my flow while I
wait for a response. I'd much rather see work being done on
documenting what we currently have in typing, rather than yet more
changes which while no doubt useful, is effectively just churn that
makes maintaining a codebase that has to support multiple Python
versions harder.

PPS Sorry if this sounds negative. TBH, I'd quite happily not use
typing if I didn't want to and stay quiet. A lot of the frustration I
see being expressed here (including my own) seems to come from the
fact that it's so difficult to actually take that sort of "I can
ignore it if I don't use it" attitude, whether that's because of
community pressure, tool requirements, or whatever.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LAR7R5MCAC74N7H7NQXJUGXG7LCOA4CB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Paul Moore]

On Tue, 30 Nov 2021 at 19:07, Brett Cannon  wrote:
>
> On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano  wrote:
>>
>> On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:
>>
>> > And to be clear, it's often very non-obvious how to annotate something
>> > - in https://github.com/pfmoore/editables I basically gave up because
>> > I couldn't work out how to write a maintainable annotation for an
>> > argument that is "a Path, or something that can be passed to the Path
>> > constructor to create a Path" (it's essentially impossible without
>> > copy/pasting the argument annotation for the Path constructor).
>
> You're after 
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike: 
> `str | PathLike[str]` (if you're only accepting string paths).

Well, it's not really. What I'm after, as I stated, is "anything that
can be passed to the Path constructor". Yes, str | PathLike[str] is
probably close enough (although why is it OK for me to prohibit bytes
paths?) but that's what I mean about copying the Path constructor's
annotations. If Path changes, I have to change my code.

This is a very common idiom:

def f(p: ???):
p = Path(p)
...

Why isn't it correspondingly straightforward to annotate?

If PathLike[str] included str, then it would be a lot easier. It's not
at all obvious to me why it doesn't (well, that's not entirely true -
it's because PathLike is an ABC, not a protocol, and it's not intended
to define "the type of objects that the Path constructor takes"). It
would still not be documented anywhere, though.

>> I thought that type inference was supposed to solve that sort of
>> problem? If the typechecker can see that an argument is passed to the
>> Path constructor, it should be able to infer that it must be the same
>> types as accepted by Path.
>
> I would change that "should" to "may". Python's dynamism makes inferencing 
> really hard.

That's fair. That's why I think it should be straightforward for the
user to explicitly say "this argument should accept the same types as
pathlib.Path does". If inference can't do it automatically, and the
user can't (easily) let the checker know that it's OK to do it, then
we're left with no easy way to express a very common pattern.

>> Aside: I'm a little disappointed in the way the typing ecosystem has
>> developed. What I understood was that we'd get type inference like ML or
>> Haskell use, so we wouldn't need to annotate *everything*, only the bits
>> needed to resolve ambiguity. But what we seem to have got is typing like
>> C, Pascal and Java, except gradual. Am I being unreasonable to be
>> disappointed? I'm not a heavy mypy user, I just dabble with it
>> occasionally, so maybe I've missed something.
>
>
> It really depends on the code base. Type checkers can make guesses based on 
> the code they have available to them, but that only works if the usage is 
> really clear and the dynamic nature of the code doesn't make things murky. 
> For instance, look at open() and how whether you opened a file with `b` or 
> not influences whether the object's methods return strings or bytes. What 
> would you expect to be inferred in that case if you didn't annotate open() 
> with overrides to specify how its arguments influence the returned object?

Personally, I'd be quite happy leaving open() as duck typed. I see
this as what Steven was getting at - the "typing ecosystem" has moved
into a situation where it's acknowledged that some typing problems are
really hard, due to Python's dynamism, and yet there's still a drive
to try to express such highly dynamic type constraints statically. And
worse still, to insist that doing so is somehow necessary.

Whatever happened to "practicality beats purity", and typing being
"gradual"? Surely annotating everything except open() is practical and
gradual?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YXVXN22TIHBT454CKHKKOCELANF73FZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 12:39, Oscar Benjamin  wrote:

> There are other open "issues" like this for SymPy where the
> presumption is that not having type hints is now to be considered a
> deficiency of the library regardless of whether the hints have any
> benefit for internal use. I don't object to adding the hints but it's
> a huge amount of work that someone would have to do and I don't want
> to add useless/inaccurate hints temporarily (as some have suggested to
> do).

This is precisely the sort of concern I have. Although it is awfully
tempting to passive-aggressively annotate everything as Any, just to
shut people up :-(

And to be clear, it's often very non-obvious how to annotate something
- in https://github.com/pfmoore/editables I basically gave up because
I couldn't work out how to write a maintainable annotation for an
argument that is "a Path, or something that can be passed to the Path
constructor to create a Path" (it's essentially impossible without
copy/pasting the argument annotation for the Path constructor). I also
spent a lot of time trying to deal with a typeshed bug. And even
worse, if I put the types in a .pyi file (which I want to do, because
I don't want the library to import typing at runtime because it's a
significant overhead for something that's supposed to be lightweight),
apparently mypy won't check them so I gain no benefit for the project
itself.

Anyway, we're *way* off topic now, and I doubt there's much that the
SC or python-dev can do to change the community view on typing,
anyway.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4DUJM7MNJSJ3TOYYFILQCG4RT4CU6DME/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 09:37, Chris Angelico  wrote:
>
> On Tue, Nov 30, 2021 at 8:19 PM Paul Moore  wrote:
> > Also, related to the question Terry raised, IMO it would be useful to
> > have a clear statement on code that *does* use type annotations, but
> > violates them at runtime. To be specific, is the following considered
> > as an error?
> >
> > >>> def muladd(x: int, y: int, z: int) -> int:
> > ... return x * (y+z)
> > ...
> > >>> muladd(3.1459, 87.33, 2.7e2)
> > 1124.124447
> >
>
> My understanding is that it's precisely as wrong, or not wrong, as this:
>
> def add(x, y):
>"""Multiply two numbers"""
> return x / y
>
> To my mind, annotations are machine-readable metadata, with no
> inherent "correctness" to them (from the language's point of view),
> other than syntactically.

That makes sense to me, and I'd support that being clearly stated as
the "official position".

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LUY2SXBGX4TQKTX6PE2FHYHABDCYY2IN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 02:52, Steve Dower  wrote:
>
> THAT'S the kind of thing that also has been happening with typing, and
> why some of us feel the need to publicly re-state things that are all
> agreed upon within this group, but are struggling to be heard over the
> public discourse on the topics. And this kind of reiteration is easier
> when our official documents (PEPs, etc.) state it explicitly.

This is very definitely the case. There's a subtle (maybe not so
subtle, actually) and increasing pressure on projects to add typing.
Often with little or no justification beyond "you should", as if
having typing is a sort of "obvious best practice". Sometimes "because
it will make it easier for your users who use typing" is given as a
justification, but while that's fair, it's also a disturbing gradual
pressure for typing to extend everywhere, manifesting by making it
feel like not adding typing is somehow "not caring about your users".

Also, related to the question Terry raised, IMO it would be useful to
have a clear statement on code that *does* use type annotations, but
violates them at runtime. To be specific, is the following considered
as an error?

>>> def muladd(x: int, y: int, z: int) -> int:
... return x * (y+z)
...
>>> muladd(3.1459, 87.33, 2.7e2)
1124.124447

If (as in the current interpreter) typing is optional, this isn't an
error. Certainly, running mypy over it fails, but the point here is
that mypy is an external tool - the Python interpreter itself (and the
language definition) allows it. I think it would be useful to have a
clear message that this is intended, and won't be altered lightly.
Remember that the "correct" annotation here is either muladd(x: Any,
y: Any, z: Any) -> Any, or muladd(x: SomeMultiplyProtocol, y:
SomeAddProtocol, z: SomeAddProtocol) -> Any, neither of which is
particularly helpful...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E36ILTCKFVW3MNFU753PKKWIFY2FWQXU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Fri, 26 Nov 2021 at 17:13, Guido van Rossum  wrote:

>> Although the more I think about it, given that I believe dataclasses
>> use eval "under the hood", the less I understand *how* it manages to
>> do that without special-case knowledge of the dataclass decorator...)
>
> Static checkers special-case the @dataclass decorator. Eric Traut has a 
> proposal to generalize this support (sorry, I'm in a rush, otherwise I'd dig 
> up the link, but it's in the typing-sig archives).

:-( That's what I suspected, but it does mean that dataclasses has a
privilege that other libraries (like attrs, I guess?) don't get.

>> I'd like to see a clearer statement from "somewhere" about how APIs
>> should use annotations at runtime, such that Python users have a much
>> clearer intuition about APIs like the dataclass one, and library
>> designers can build their APIs based on a clear "common understanding"
>> of what to expect when annotations are used.
>
> Note that @dataclass itself is very careful not to use the annotations, it 
> only looks for their *presence*. With one exception for ClassVar.

Understood. What I'm suggesting is that it would be good to have a
clear "common understanding" about whether libraries should be careful
like this, or whether it's OK to base runtime behaviour on type
annotations. And if it is OK, then what are good patterns of design
and behaviour? This is where the proposal to store annotations as
strings hit issues, because it appears to take the view that libraries
*shouldn't* be looking at the actual types specified by annotations
(or maybe that they should only do so via something like
`typing.get_type_hints`). There are other subtleties here (runtime
code needs to deal with the fact that int and "int" should be treated
the same) that there's no guidance on, again possibly because no-one
is really considering that use case.

Paul

PS I've never written code myself that does runtime introspection of
type annotations - so it's quite possible that there *is* guidance
that I've just missed. But it wasn't obvious to me from a quick search
- the "introspection helpers" section of the typing module docs is
pretty basic...
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y4WTMMHZ5XR6PCPKRP4FTIX4R2YJCS7L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Thu, 25 Nov 2021 at 21:45, Christopher Barker  wrote:
> The issue is that, intended or not, typing is making it's way into Python 
> culture. As an instructor of beginning python users, I am unsure at this 
> point when to introduce type annotations.
>
> What is their role? Up to today, I have treated them as an advanced feature, 
> useful for "complex codebases". But there are any number of examples 
> springing up on the internet, to the point where many students now think they 
> are "best practice", if not actually required.

Agreed this is somewhat OT, but I also think the messaging around
annotations needs to be reviewed. I suspect that students will also
not clearly understand the fact that annotations aren't checked at
runtime. And IMO this is particularly difficult to get across in the
case of non-typechecker uses of annotations.

The following, for example, is extremely non-intuitive to me:

>>> from dataclasses import dataclass
>>>
>>> @dataclass
... class A:
...   n: int
...
>>> a = A("didn't expect this!")
>>> a
A(n="didn't expect this!")

Even though the "typing is optional" message is well-understood, my
instincts still lead me to expect that declaring the type of n will
result in a runtime type-check, in the constructor at least. (If I
check the code with mypy, it does raise an error, so that's good.
Although the more I think about it, given that I believe dataclasses
use eval "under the hood", the less I understand *how* it manages to
do that without special-case knowledge of the dataclass decorator...)

I'd like to see a clearer statement from "somewhere" about how APIs
should use annotations at runtime, such that Python users have a much
clearer intuition about APIs like the dataclass one, and library
designers can build their APIs based on a clear "common understanding"
of what to expect when annotations are used.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NBE7YL6EBVBB3NQ33DJI4KY3NSYL2Q5W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Fri, 26 Nov 2021 at 05:14, Guido van Rossum  wrote:
>
> My memory is also hazy, but I'm quite sure that *in my mind* annotations were 
> intended as a compromise between conflicting proposals for *typing*. We 
> didn't have agreement on the syntax or semantics, but we did know we wanted 
> to do something with types eventually.

More hazy memories here, but I think the original proposal left open
the possibility of annotations not being types at all - for example,
being docstrings for the arguments, or option names for a "function
call to CLI" tool, etc. I think that some libraries took this approach
(in particular the "CLI builder" idea). At some point (probably around
PEP 484) it became clear that the expectation was that annotations
would be *types*, and in particular would be the expected/intended
type of the annotated value. I think there was some discussion at that
point about "what do we do about the CLI builders" but the conclusion
was that they had relatively low adoption rates, and could adapt (or
more accurately, abandon the annotation approach). That was the point,
in my recollection, where the "annotations are for types" principle
was established. However, I feel that many people got (or were given)
the impression that static type checkers were the core use case
(probably as a result of the emphasis on the message that "type
annotations will never be mandatory"), for better or worse.

I think that by now, it's long been understood by most people that
annotations are types. This may actually be one reason why people are
so uncomfortable with the idea of stringified annotations, because it
violates that assumption - personally, I have the same discomfort
about using explicit string annotations for forward references, it
feels like I'm not declaring a "proper type".

If what I say above is right, the debate here isn't about whether
annotations "are for types", but rather about whether reading the
types in annotations and using them to affect behaviour *at runtime*
is a legitimate use of annotations. That is the use case that
stringifying annotations makes more difficult, and which doesn't seem
to have a strong enough voice in the direction of typing proposals. I
lurk on the typing-sig, and from an outsider's perspective, the
participants seem to be almost entirely designers or heavy users of
static type checkers. That gives a certain emphasis to the proposals
coming from that group.

I'd therefore interpret Barry's plea as being for *anyone* with a use
for annotations to provide their feedback (at least, anyone who
accepts that annotations are types), with particular emphasis on
people who want to use the types declared in annotations to affect
runtime behaviour, as that's the most under-represented group at the
moment (and it's not clear whether it's under-represented because
there aren't many such uses, or because the users aren't being heard
from).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5N5GTSWQPVALCTJDWZ4G4JSBPDS6324V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Sun, 21 Nov 2021 at 07:50, Christopher Barker  wrote:
>
> TL;DR:
>
> Annotations can be, and are, used for other things than "typing". I just 
> noticed that PEP 563 apparently deprecated those other uses (well, sort of: 
> "uses for annotations incompatible with the aforementioned PEPs should be 
> considered deprecated"), but if the SC is reconsidering PEP 563, then it 
> would be nice to be clear about whether non-typing uses of annotations are 
> indeed deprecated. If not, then the challenge is to come up with a way 
> forward that not only supports both static and dynamic typing, but also other 
> potentially arbitrary use cases.

I agree with the points made in this post. It's becoming harder and
harder for people not particularly interested in static typing to
simply ignore it, and any use of annotations to affect runtime
behaviour is in a weird grey area. And as a library author, I'm now
finding that I'm getting requests to add typing to my code "for my
users" (i.e., using types is no longer just a choice I make for my
project, it's an API design issue).

So I too would appreciate clarity on where annotations, and more
generally typing, stand as Python language features¹.

Paul

¹ In particular, typing features seem to change so rapidly that
supporting a wide range of Python versions is a real pain (in
practical terms - not getting syntax errors is straightforward, but
writing type annotations that work well across versions very
definitely isn't, at least for someone who doesn't really care about
static typing).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AMGOAG22UJXCFPQ2QCGH57RNB3UTAX7H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Having Sorted Containers in stdlib?

[Paul Moore]

On Thu, 11 Nov 2021 at 11:51, Antoine Pitrou  wrote:
>
> On Wed, 10 Nov 2021 21:12:17 -0600
> Tim Peters  wrote:
> > [Bob Fang ]
> > > This is a modest proposal to consider having sorted containers
> > > (http://www.grantjenks.com/docs/sortedcontainers/) in standard library.
> >
> > +1 from me, but if and only if Grant Jenks (its author) wants that too.
> >
> > It's first-rate code in all respects, including that it's a fine
> > example _of_ Python programming (it's not written in C - in Python).
>
> Agreed with Tim.  This is a perfect example of some basic and perennial
> facility that would fit very well in the stdlib.

I agree as well. Is anyone interested enough to ask the library author
if he supports doing this? That seems to be the main unanswered
question here.

But if anyone wants to argue the "the stdlib should be shrinking, not
growing" position, I suggest they do so *before* someone reaches out
to the module author. No point in us making the suggestion and then
being forced to withdraw it.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5SURNB4C5FGJ6LSXUPVW2EFP22ERKSGB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proposal: Allow non-default after default arguments

[Paul Moore]

On Tue, 9 Nov 2021 at 10:39, Sebastian Rittau  wrote:

> This might be better API design (although I don't think Python should be
> opinionated about this outside the standard library), but this still
> leaves the API change example and the very real problem of @overloads
> unsolved.

You can handle this using *args and/or **kwargs, so Python does
support this API design if you want to use it. I disagree, however,
with the statement that Python shouldn't be "opinionated" about this.
It's not a matter of being opinionated, IMO, it's about whether Python
has to have explicit language support for every possible use case that
anyone comes up with. In this case, the situation seems rare enough,
and the API design is sufficiently unusual (which is a polite way of
me saying that I think it's a bad design...), that I think it's
entirely reasonable for Python to not support it outside of the
generic *args/**kwargs machinery.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YO35XE6JHYNGQ4JHX3ZI4YBKXJ5XTG4O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: containment and the empty container

[Paul Moore]

On Tue, 9 Nov 2021 at 06:31, Ethan Furman  wrote:

(IMO, Steven D'Aprano's comment above, which I don't think you have
responded to yet, is the clearest summary of the options here. I agree
with what he says, but just wanted to clarify one point below).

> I've had a couple people tell me that they think of flags as sets, and use 
> set theory / set behavior to understand how
> flags and groups of flags should interact.  My main concern is whether that 
> is the only correct way to look at flags,
> and whether the behavior that seems logical to me with regards to, for 
> example, RegexFlag(0), is wrong, unusual, or
> perfectly natural.

Looking at flags as sets, you need to make a very clear distinction¹
between a *group* of flag values (a set) and a *single* flag value (an
element). The operation "in" on sets tests if an *element* is a member
of a *set*. In the examples you are giving, you seem to be confusing
sets with elements - if you have flags 1 (0b001), 2 (0b010) and 4
(0b100), then asking if 3 is "in" 1|2|4 makes no sense, because 3 is a
*set* of flags, and sets don't go on the LHS of the "in" operator.
Asking if 1 is "in" 3 makes some level of sense (element in set) and
is true, but it looks much less convincing when expressed as numbers
rather than named values (where the names can be chosen to make 1
"look like" a value and 3 "look like" a set).

¹ Pedantry digression - in pure set theory, there are *only* sets, and
sets are members of other sets. But this is largely a technical detail
(and even then, some set theories have the idea of "levels" of sets,
which formalises the whole set vs element idea). Real-world intuitions
about sets tend to have sets and elements as different concepts.

> The way I see it, the following should hold
>
>  empty_flag = RegexFlag(0)
>  any_case = RegexFlag.IGNORECASE
>  any_case_on_any_line = RegexFlag.IGNORECASE | RegexFlag.MULTILINE
>
>  any_case in empty_flag is False
>  any_case_on_any_line in empty_flag is False
>
>  empty_flag in any_case is False
>  empty_flag in any_case_on_any_line is False

You seem to be confusing "in" with "is a subset of" here. Strong -1 on
using "in" to mean "is a subset", as that breaks the pattern used for
every other type in Python. None of the examples you give here make
any sense to me, either in terms of a "natural reading" nor in terms
of "flags are like sets and their elements". You also seem to be
confusing a flag value with the 1-element *set* of values containing
just that value².

The use of the name empty_flag for the value RegexFlag(0) makes things
even more confusing for me, as it's emphatically *not* a single flag,
but a collection (specifically the empty collection) of flags.

If I had to think of the flag type strictly as a set type, I'd think
of all the values as sets of flags (even RegexFlag.IGNORECASE would be
"a set of one flag") and the operation you're talking about here would
then be "subset" - which I'd either spell as "<=" or I wouldn't use an
operator for *at all*. I could just barely deal with thinking of
individual bits as having a dual "set or element" nature, which would
make "individual_flag in flag_set" make sense to me, but it *only*
makes sense if the LHS has exactly one bit set. And I think this is
where the confusion comes in - we typically don't think of operators
as only being meaningful for certain values of one operand (well,
there's 1/0, so I guess we could have "Flag(0) in Flag(1)|Flag(2)"
raise ValueError, by analogy with ZeroDivisionError, but that doesn't
seem helpful in any practical situation).

Overall, my view is that we shouldn't support "in" on enum.Flag types
at all, as the potential for confusion is too high. I'm -0 on having
an explicit "is subset" operator, but if we do, it should be named
"<=", not "in". I fear that any attempt to document an "is subset"
operator would get bogged down in confusion over whether a flag value
is a value or a set of values (or would completely confuse people with
a background in set theory, by making it look like "element is subset
of set" was somehow a thing).

² Pedantry digression 2 - set theories that allow values that are
equal to the set containing just that value are possible (such values
are called "Quine atoms" - see
https://math.stackexchange.com/questions/2389726/can-one-element-set-be-considered-equal-to-its-element)
but such theories are not mainstream, and are not as intuitive as
you'd hope - for example, "RegexFlag.IGNORECASE in
RegexFlag.IGNORECASE" just seems weird to me...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EHPB5775QZOWUIS3LJPISQ6FN4HBRJUG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Oh look, I've been subscribed to python/issues-test-2 notifications again

[Paul Moore]

I've had about 5 or 6 of them.
Paul

On Thu, 4 Nov 2021 at 19:22, Brett Cannon  wrote:
>
> What notification? (I fully admit I may not have gotten one due to some team 
> I'm in, but I have no such notification if it happened recently.)
>
> On Thu, Nov 4, 2021 at 12:16 AM Larry Hastings  wrote:
>>
>>
>> I guess this is part of the migration from bpo to GitHub issues?  Maybe the 
>> initial work could be done in a private repo, to cut down on the spurious 
>> email notifications to literally everybody subscribed to cpython?  Which is 
>> a lot of people.
>>
>>
>> /arry
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/H5KW6GRHIF2VWOGNRH5367WB3K2GPARO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/OJYGOHGS27CUY7KWHE5PEC2CKYH6M5PI/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JGPFO6R4AUDTF7LG6SSFDC6Y6GRTA66S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pre-PEP: Unicode Security Considerations for Python

[Paul Moore]

On Wed, 3 Nov 2021 at 10:11, Marc-Andre Lemburg  wrote:
> I don't think limiting the source code encoding is the right approach
> to making code more secure. Instead, tooling has to be used to detect
> potentially malicious code points in code.

+1

Discussing "making code more secure" without being clear on what the
threat model is, is always going to be inconclusive. In this case, I
believe the threat model is "an untrusted 3rd party submitting a PR
which potentially contains malicious code to a Python project". For
that threat, I think the correct approach is for core Python to
promote awareness (via this PEP and maybe something in the docs
themselves) and for projects to implement appropriate code checks that
are run against all PRs to flag this sort of issue.

What threat can't be addressed at a per-project level, but *can* be
addressed in core Python (without triggering so many false positives
that people are trained to ignore the warnings or work around the
prohibitions, defeating the purpose of the change)?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FQ42C66BVCE6AQFSP4J6V6ERS4VV44MK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Paul Moore]

On Tue, 19 Oct 2021 at 00:55, Guido van Rossum  wrote:
>
> I should have added that I also don't feel I want to go at bat to fight for 
> this PEP. I do observe that it looks like the folks used to building large 
> systems (in Python or other languages) don't seem to like it, while it seems 
> to appeal to folks writing simpler code (the abundant majority of Python 
> users, but not of Python core devs). I worry that the experienced folks may 
> perhaps be a little too eager to protect newbies from shooting themselves in 
> the foot.

Possibly. But in *my* case, I'm not arguing from the position of a
large system builder, but from that of a script writer (albeit an
experienced one) and code maintainer (usually of my own code).

I find y = config?.get("handler")?.get("parameters")?.get("y")
unreadable and confusing, and I'd probably advise strongly against it
if someone ever showed me code containing it. I see y =
get_config(config, "handler", "parameters", "y") as *far* more
readable and expressing the intent more directly.

Yes, I find Steve's arguments persuasive, but they are not the ones
I'd be concerned with when advising a newcomer. Rather I'd be saying
"do you see how using a named function expresses your intent better?"
and "do you see how writing a small function hides the messiness of
checking for None so that your main code is cleaner?"

*Shrug* I guess I just don't understand how people can look at a
string of ?.get() and see it as readable and obvious :-(

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QOWSJXN3WE7WLUYB22HZNOZJ6GGGRWVF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 505 (None-aware operators) for Python 3.11

[Paul Moore]

On Mon, 18 Oct 2021 at 19:29, Guido van Rossum  wrote:

> where the convention is that keys at any level may be omitted altogether and 
> config itself may be NOne, then to safely access the value of 
> config["handler"]["parameters"]["y"] we would have to write
>
> y = None  # Default
> if config is not None:
>   handler = config.get("handler")
>   if handler is not None:
> parameters = handler.get("parameters")
> if parameters is not None:
>   y = parameters.get("y")
>
> This kind of pattern (and all the various other ways of writing it, e.g. 
> using the walrus or passing {} as the second argument to dict.get()) can be 
> *very* common if that's the kind of data you're given and that's the kind of 
> app you have to write, and you can't control the format of the data.
>
> Using ?. this can be written as
>
> y = config?.get("handler")?.get("parameters")?.get("y")
>
> More examples are in PEP 505 itself, see 
> https://www.python.org/dev/peps/pep-0505/#examples

For this particular usage, I'd much rather have a functional API, like

y = get_config(config, "handler", "parameters", "y")

I understand that writing many helpers like that nested_get is a
chore, and having language support for the operation avoids that chore
- but I'm with Steve in not wanting to see the ?.get() pattern become
"idiomatic Python" as it encourages people *not* to design APIs like
nested_get that allow the user to not even be aware of all that
behind-the-scenes complexity. Sure, you could argue that the ?.
operator makes it easier to write something like get_config, but it's
not *that* hard:

def get_config(config, *keys):
value = config
for key in keys:
if value is None:
break
value = value.get(key)
return value

And the problem with the ?.get style is that it doesn't hide anything
- what if you want/need to change your config data structure (because
the JSON you're reading changes its layout, say)? Without
encapsulation, you can't. And if it's *that* common, adding a stdlib
function or a new dict method is also an option, which doesn't need a
language feature and demonstrates good (IMO) API design for people to
copy.

Anyway, much like Steve, I don't expect to spend a lot of time
fighting this proposal. But I will be sad if it gets accepted, and
even more so if ?. becomes idiomatic in user code.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/55SOK3J6YRIOEBX47INLLTAVUZRMD57Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Paul Moore]

On Fri, 8 Oct 2021 at 16:29, Jelle Zijlstra  wrote:
> That's one of the differences between the proposals discussed here. The basic 
> proposal Pradeep pointed out would not support named arguments, the more 
> complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) 
> would support it.
>
> Pradeep has done some empirical analysis that shows that in practice it is 
> not very common to use such types, though.

I believe resolvelib (used by pip) makes a point of always calling
callbacks with named arguments, to ensure compatibility. I thought PEP
517 hooks also required callers to use keywords, but when I checked
the PEP, apparently not. I think with hindsight that we should have,
though.

But yes, I can imagine that simpler callbacks (like key= in sort)
would be more common, and would be fine with positional arguments.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EUXX7JBUXTNYZWKJ2XG7BXJST65DHAVI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Paul Moore]

On Fri, 8 Oct 2021 at 03:45, S Pradeep Kumar  wrote:

> Typing-sig has been discussing user-friendly syntax for the type used to 
> represent callables. [1] Since this affects the Python language syntax, we 
> wanted to get some high-level feedback from you before putting up a detailed 
> PEP.

Disclaimer: I personally find that use of complex type annotations
like Callable make code unreadable for the *human* reader, as compared
to out of line information like a clearly-written comment or
documentation. Therefore I'd probably be extremely reluctant to use
this new syntax in real-life projects, regardless of the form that it
takes. However, I do have some experience working on projects that use
type annotations like this for callbacks, so I will answer in spite of
my personal reservations.

> **Question 1**: Are there concerns we should keep in mind about such a syntax 
> proposal?

In my experience, a common pattern for functions that take non-trivial
callbacks is to call them with named arguments rather than positional,
in order to preserve future compatibility. So I would like callback
annotations to be able to express the type "a function returning X,
taking arguments `arg1` of type Y and `arg2` of type Z". The lack of
keyword argument support in this proposal makes it unusable for APIs
like this.

Conversely, for callbacks with single arguments like f(cb: int->int)
-> int, in my experience a lot of uses are lambda expressions, so
something like f(lambda x: x+1). I would be a strong -1 on having to
add types to lambda expressions (they are ugly enough already) so how
does this interact with the annotation? Are type checkers able to
correctly infer the type of lambda x: x+1, or would the whole
expression end up being untyped? Similarly with functools.partial -
would that correctly match the type? If common uses of callbacks end
up being untyped in practice, that drastically reduces the value of
the new syntax. (Note that this point is equally applicable to
Callable - so I guess it's not so much about the new syntax as it is
about whether annotating callbacks is sufficiently useful to be
*worth* improving).

Also, note that I automatically used a type of int->int up there. As
someone else asked, is that form allowed, or is it required to be
(int)->int? In my view, if we require the latter, I expect there will
be a *lot* of people making that mistake and having to correct it.

> **Question 2**: Do you have preferences either way? Do we propose (1) alone 
> or (1) + (2)?

See above. My experience is that using named arguments for callbacks
is a useful approach, and should probably be encouraged rather than
made difficult, so I'd say that named argument support is important.
Having said that, I think that making simple types like int->int or
(int, int)->int more awkward to write would be a disadvantage - they
are used a *lot* in things like the `key` argument of `sort`. So I'm
-1 on things like (for example) requiring a colon as in (:int, :int)
-> int.

But to reiterate, I'm probably not likely to ever be a significant
consumer of this feature, so consider the above with that in mind ;-)

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ANGLWMP4X7AIBFI5SII7RBCOHRIZZ7TH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Does ensurepip still have to include a copy of setuptools?

[Paul Moore]

On Mon, 4 Oct 2021 at 08:21, Antoine Pitrou  wrote:
>
> On Sat, 2 Oct 2021 13:27:03 +0100
> Paul Moore  wrote:
> > On Sat, 2 Oct 2021 at 12:20, Thomas Grainger  wrote:
> > >
> > > I raised an issue about this: https://github.com/pypa/pip/issues/10530
> >
> > I agree with the comment made on that issue - this isn't the right way
> > to handle the problem. We need to encourage projects to opt into the
> > new approach and remove the legacy path once it's no longer needed. We
> > should *not* maintain the "old style" approach indefinitely, hiding
> > the fact that it's no longer the correct approach by having some sort
> > of "auto convert" logic in the tools.
>
> Can you explain what the "old style" approach is here?  I would hope
> for the "old style" approach to be deprecated (with a *visible*
> warning message) for at least 2 years before it is removed.

I'm talking about projects adopting a `pyproject.toml` configuration
file that specifies the build backend to use (in this case
setuptools). The `pyproject.toml` style was standardised in PEP 517
about 4 years ago now, and projects have been gradually adopting it.
As you say, there's a long tail of projects who have no immediate need
to switch, but we're working on smoothing that transition as much as
we can.

Deprecation is a complex process, and not really a python-dev
question, but for the record, pip (and PEP 517) have a mechanism for
using the new-style hooks even for older projects that haven't adopted
it. That, plus a "build isolation" mechanism, allows pip to work even
if setuptools is not present. We're transitioning to making that the
default behaviour, but that process isn't yet complete. Although even
when it *is* complete, we may have options allowing use of the old
behaviour (`--no-build-isolation, --no-use-pep517) for some time after
that.

Regarding warning when the old `setup.py` mechanism is used instead of
the new PEP 517 hooks, that's a matter for setuptools to decide, and I
can't speak for them. It's also not relevant to when ensurepip drops
inclusion of setuptools, as the ensurepip requirement is only that
*pip* no longer needs setuptools, and as I said, we're hoping to make
that mostly transparent. We *might* also be able to add a warning in
pip, to catch the case where setuptools *doesn't* have the warning,
but honestly we will probably be mostly OK from our side of things
with just advising the user to install setuptools manually in the
(increasingly rare) cases where we can't work out to install it
automatically.

> It is nice that well-maintained packages with lots of contributors get
> frequent releases and keep up with the pace of changes in the packaging
> ecosystem, but please don't forget that there's a long tail of packages
> that are updated infrequently and but still work properly and perform
> an important function for some parts of the user base.

We (the packaging community) are *extremely* aware of this, yes. If
you're interested in helping out, then these sorts of discussions
happen on the packaging area in Discourse, and (for project-specific
items) on the pip and setuptools trackers. We'd love more help there -
packaging for Python is critically under-resourced! - so please feel
welcome to come along and join in the work :-)

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M6CZYUEKKNZC6ANSAULKP7JYWUJY6V63/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Paul Moore]

On Mon, 4 Oct 2021 at 07:16, Greg Ewing  wrote:
>
> On 4/10/21 6:23 pm, Guido van Rossum wrote:
> > On Sun, Oct 3, 2021 at 9:20 PM Jonathan Goble  > > wrote:
> >
> > Therefore my vote is for requiring `except* E` and keeping `except
> > *E` as a SyntaxError.
> >
> > You can't do that with our current lexer+parser.
>
> I don't think it would be desirable in any case. The separation of
> tokens into alphanumeric and non-alphanumeric is deeply embedded in
> every Python programmer's brain by now, and we shouldn't mess with
> it.

Agreed. Having "except*" be a single token, distinguished from the
pair of tokens "except" "*" only by the presence of whitespace, would
be extremely confusing.

And yes, I am aware that 3.as_integer_ratio() and 3.
as_integer_ratio() are syntax errors, whereas 3 .as_integer_ratio()
and 3 . as_integer_ratio() are valid. IMO, that's *also* very
confusing, and serves as a warning to not do that again, and not as an
example of how it's OK and we can do more of that...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IKWKS6VYWFQ4XEXJ4XFYBLPRPXATKGGL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 except* formatting

[Paul Moore]

On Sun, 3 Oct 2021 at 16:55, Irit Katriel via Python-Dev
 wrote:
>
> We wonder if people have a view on which of the following is clearer/better:
>
> 1. except *E as e:  //  except *(E1, E2) as e:
> 2. except* E as e:  //  except* (E1, E2) as e:
>
> (The difference is in the whitespace around the *).

I prefer (1). I never liked C declarations where the * was attached to
the type rather than the variable, and I have the same dislike here.

> At the moment * is a separate token so both are allowed, but we could change 
> that (e.g., make except* a token), and in any case we need to settle on a 
> convention that we use in documentation, etc.

Having said the above, it's a matter of taste/preference, so I think
that allowing both is the correct thing to do.

> It is also not too late to opt for a completely different syntax if a better 
> one is suggested.

Let's stick with "except *". It doesn't seem productive to have
another round of bikeshedding at this point, unless there's a really
compelling technical reason (i.e., something significantly more than
mere bikeshedding).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BZ55ZMS5S5E7KPNP7AR7N2BSA35KVKF3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Does ensurepip still have to include a copy of setuptools?

[Paul Moore]

On Sat, 2 Oct 2021 at 12:20, Thomas Grainger  wrote:
>
> I raised an issue about this: https://github.com/pypa/pip/issues/10530

I agree with the comment made on that issue - this isn't the right way
to handle the problem. We need to encourage projects to opt into the
new approach and remove the legacy path once it's no longer needed. We
should *not* maintain the "old style" approach indefinitely, hiding
the fact that it's no longer the correct approach by having some sort
of "auto convert" logic in the tools. Doing that has the *opposite*
effect to what we're trying to achieve - adoption of cleaner modern
approaches will take *longer*, because we're actively allowing
projects to continue using their existing approach with no
consequences.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YACJ6OZ2PQ5DLGZUE2IJLYKR6M3SKAP7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Does ensurepip still have to include a copy of setuptools?

[Paul Moore]

On Sat, 2 Oct 2021 at 03:27, Illia Volochii  wrote:
>
> Hi everyone,
>
> ensurepip includes private copies of pip and setuptools. But PEP 453 states 
> that "once pip is able to run pip install --upgrade pip without needing 
> setuptools installed first, then the private copy of setuptools will be 
> removed from ensurepip in subsequent CPython releases."
> https://www.python.org/dev/peps/pep-0453/#automatic-installation-of-setuptools

Interesting. Pip does not need setuptools installed to upgrade itself,
so a strict reading of the PEP would seem to imply that we need to
remove setuptools, as you say. However, looking at things more
practically, pip still needs setuptools to do "legacy" installs of
source distributions (when the project does not include a
`pyproject.toml`) and for editable installs of setuptools-based
projects. If we stopped shipping setuptools as part of ensurepip, I
imagine people would complain that we'd "broken" things. Telling them
that it's not broken, all they need to do is `pip install setuptools`,
and we only ever promised that the supplied pip could be used to
bootstrap a full environment, doesn't seem likely to go down well IMO.

(Technically, some aspects of pip don't work, or fall back to "legacy"
code paths, if the `wheel` project isn't installed, so ensurepip
"needs" wheel in the same sense as it "needs" setuptools, but the
breakage is less significant, and people who care are used to the
current situation and know what to do. So yes, that argues we could do
the same to setuptools, it's just a bigger impact.)

> At the moment pip itself includes a needed part of setuptools.
> https://github.com/pypa/pip/tree/9c474d4862907ae220ced0fcdbd76660955ff732/src/pip/_vendor/pkg_resources

That's internal to pip, and the pip code that uses that, does not need
an externally-supplied setuptools.

> I experimented with modifying ensurepip in the main branch not to install 
> setuptools, and then used it to install pip. It worked fine.
> Then I run `./python -m pip install --upgrade pip`, and it upgraded pip 
> successfully.
>
> Does this mean that we can drop the copy of setuptools?

IMO, it's too early to consider dropping setuptools, notwithstanding
what a strict reading of the PEP says. When pip has removed more of
the "legacy" code paths, this situation could change, but we're not
there yet.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AM7ATX4IWLNXKG54Z34GYZ2D7RJWQUNC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Paul Moore]

On Tue, 28 Sept 2021 at 15:33, Eric Snow  wrote:

> > It means that the site module module can no longer be "customized" by
> > modifying directly the site.py file (inject a path in PYTHONPATH env
> > var where the customized site.py lives). But there is already a
> > supported way to customize the site module: create a module named
> > "sitecustomize" or "usercustomizer". I recall that virtualenv likes to
> > override stdlib site.py with its own code. tox uses virtualenv by
> > default. Someone should check if freezing site doesn't break
> > virtualenv and tox, since they seem to be popular in Python. The venv
> > doesn't need to override site.py and tox can use venv if I recall
> > correctly.
> >
> > If site.py customization is too popular, I would suggest to not freeze
> > this one, until the community stops doing that.
>
> Good point.  I'll look into that.

I don't believe virtualenv ships its own site.py these days. That was
a historical thing, and was always a pain point, but when virtualenv
got rewritten I'm almost certain we stopped doing it.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C5J763VQV423T4KIA4WRDIEJUQ3423FA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should the definition of an "(async) iterator" include __iter__?

[Paul Moore]

On Thu, 16 Sept 2021 at 01:30, Chris Barker via Python-Dev
 wrote:
> """
>  "[i]terators are required to have an __iter__() method" which neither `for` 
> nor `iter()` actually enforce.
> """
>
> I'm confused -- as far as I can tell `for` does enforce this -- well, it 
> doesn't enforce it, but it does require it, which is the same thing, yes? But 
> does it need to?

for enforces that *iterables* have an __iter__ method, not
*iterators*. (for takes an iterable, not an iterator, and uses
__iter__ to *get* an iterator from it).

The debate here is (I think!) whether an *iterator* that is not also
an *iterable* is a valid iterator.

IMO it is valid (because that's what the definitions say, basically)
but it may not be *useful* in certain circumstances, and it definitely
may not be *expected* (because nearly all iterators are iterables).
"Broken" is a strong word to use, though, and that might be why the
debate is continuing this long...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UIKBKWT5G4ME2LVZ3W6RYRK5ESNBEZBQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: A better way to freeze modules

[Paul Moore]

On Fri, 3 Sept 2021 at 10:29, Simon Cross  wrote:
>
> Hi Gregory,
>
> I think adding a meta path importer that reads from a standard
> optimized format could be a great addition.

I think the biggest open question would be "what benefits does this
have over the existing zipimport?" Maybe it could be a little faster?
But would the downside of it not being possible to manage the format
with existing standard tools outweigh that?

A clear description of how to decide which is the most appropriate to
use in a given situation between the new format and a zipfile would be
a benefit here.

> As you mentioned in your email, this is a big detour from the current
> start-up performance work, so I think practically the people working
> on performance are unlikely to take a detour from their detour right
> now.

Agreed, it would probably have to be an independent development
initially. If it delivers better performance, then switching the
startup work to use it would give a second set of performance
improvements, which no-one is going to object to. Similarly, if it's
simpler to manage, then the maintainability benefits could justify
switching over.

> * Ask if there are any Python core developers who would be willing to
> look at the early stages of the code and/or PEP that you might produce
> in the next couple of steps. Perhaps also ask on one of the packaging
> mailing lists. If you get others involved as reviewers or contributors
> from the start, convincing them later that it is a good idea will be
> much easier. :)

I'd be willing to look. I'm more interested in the design at this
stage than in looking at code, as it's awfully easy to develop
something that ends up being a "solution looking for a problem", so a
solid case for having a general solution would be important for me.

> * Write the meta path importer in a separate package (it sounds like
> you've already done a lot of the work and gained a lot of
> understanding of the issues while writing PyOxidizer!)

This is the key thing, though. The import machinery allows new
importers to be written as standalone modules, so I'd strongly
recommend that the proposed format/importer gets developed as a PyPI
module initially, with the PEP then being simply a proposal that the
module gets added to the stdlib and/or built into the interpreter. The
key argument would be bootstrapping, IMO. I would definitely expect
interest in something like this to be lower if it's an external module
(needing a dependency to load your other dependencies is suboptimal).
Conversely, though, if no-one shows any interest in a PyPI version of
this idea, that would strongly imply that it's not as useful in
practice as you'd hoped.

In particular, I'd involve the maintainers of pyinstaller in the
design. If a new "frozen module importer" mechanism isn't of interest
to them, it's probably not going to get the necessary support to be
worth adding to the stdlib.

> * Write a PEP.
>
> It seems to me that PEPs that come with an implementation and the
> support of a few existing core developers have a much less painful PEP
> review process.

Agreed. In particular, existing code with a clearly demonstrated user
base only has to persuade people that being in the core is important.
Most proposals that I've seen which could be developed as a PyPI
module never get anywhere because it turns out no-one is willing to do
the work. You don't need to be a core developer to write a PyPI
module, so if no-one has done that, it's likely to be either because
the implementation needs tight integration into the core, or because
nobody is actually as interested in the issue as you thought...

On a personal note, I love the flexibility of Python's import system,
and I've always wanted to write importers for additional storage
formats (import from a sqlite database, for instance). But I've never
actually done so, because a zipfile is basically always sufficient for
any practical use case I've had. One day I hope to find a real use
case, though :-)

> Thank you for writing PyOxidizer and offering some of your time to
> help make Python itself better.

+1

Paul.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2THOX327G2SBUBNC4CEFK2JOH7VICFHC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: A better way to freeze modules

[Paul Moore]

My quick reaction was somewhat different - it would be a great idea, but
it’s entirely possible to implement this outside the stdlib as a 3rd party
module. So the fact that no-one has yet done so means there’s less general
interest than the OP is suggesting.

And from my experience, the reason for that is that zipimport is almost
always sufficient. That’s what tools like pyinstaller use, for example.

Paul

On Fri, 3 Sep 2021 at 06:25, Guido van Rossum  wrote:

> Quick reaction: This feels like a bait and switch to me. Also, there are
> many advantages to using a standard format like zip (many formats are
> really zip with some conventions). Finally, the bytecode format you are
> using is “marshal”, and is fully portable — as is zip.
>
> On Thu, Sep 2, 2021 at 21:44 Gregory Szorc 
> wrote:
>
>> Over in https://bugs.python.org/issue45020 there is some exciting work
>> around expanding the use of the frozen importer to speed up Python
>> interpreter startup. I wholeheartedly support the effort and don't want to
>> discourage progress in this area.
>>
>> Simultaneously, I've been down this path before with PyOxidizer and feel
>> like I have some insight to share.
>>
>> I don't think I'll be offending anyone by saying the existing CPython
>> frozen importer is quite primitive in terms of functionality: it does the
>> minimum it needs to do to support importing module bytecode embedded in the
>> interpreter binary [for purposes of bootstrapping the Python-based
>> importlib modules]. The C struct representing frozen modules is literally
>> just the module name and a pointer to a sized buffer containing bytecode.
>>
>> In issue45020 there is talk of enhancing the functionality of the frozen
>> importer to support its potential broader use. For example, setting
>> __file__ or exposing .__loader__.get_source(). I support the overall
>> initiative.
>>
>> However, introducing enhanced functionality of the frozen importer will
>> at the C level require either:
>>
>> a) backwards incompatible changes to the C API to support additional
>> metadata on frozen modules (or at the very least a supplementary API that
>> fragments what a "frozen" module is).
>> b) CPython only hacks to support additional functionality for "freezing"
>> the standard library for purposes of speeding up startup.
>>
>> I'm not a CPython core developer, but neither "a" nor "b" seem ideal to
>> me. "a" is backwards incompatible. "b" seems like a stop-gap solution until
>> a more generic version is available outside the CPython standard library.
>>
>> From my experience with PyOxidizer and software in general, here is what
>> I think is going to happen:
>>
>> 1. CPython enhances the frozen importer to be usable in more situations.
>> 2. Python programmers realize this solution has performance and
>> ease-of-distribution wins and want to use it more.
>> 3. Limitations in the frozen importer are found. Bugs are reported.
>> Feature requests are made.
>> 4. The frozen importer keeps getting incrementally extended or Python
>> developers grow frustrated that its enhancements are only available to the
>> standard library. You end up slowly reimplementing the importing mechanism
>> in C (remember Python 2?) or disappoint users.
>>
>> Rather than extending the frozen importer, I would suggest considering an
>> alternative solution that is far more useful to the long-term success of
>> Python: I would consider building a fully-featured, generic importer that
>> is capable of importing modules and resource data from a well-defined and
>> portable serialization format / data structure that isn't defined by C
>> structs and APIs.
>>
>> Instead of defining module bytecode (and possible additional minimal
>> metadata) in C structs in a frozen modules array (or an equivalent C API),
>> what if we instead defined a serialization format for representing the
>> contents of loadable Python data (module source, module bytecode, resource
>> files, extension module library data, etc)? We could then point the Python
>> interpreter at instances of this data structure (in memory or in files) so
>> it could import/load the resources within using a meta path importer.
>>
>> What if this serialization format were designed so that it was extremely
>> efficient to parse and imports could be serviced with the same trivially
>> minimal overhead that the frozen importer currently has? We could embed
>> these data structures in produced binaries and achieve the same desirable
>> results we'll be getting in issue45020 all while delivering a more generic
>> solution.
>>
>> What if this serialization format were portable across machines? The
>> entire Python ecosystem could leverage it as a container format for
>> distributing Python resources. Rather than splatting dozens or hundreds of
>> files on the filesystem, you could write a single file with all of a
>> package's resources. Bugs around filesystem implementation details such as
>> case (in)sensitivity and Unicode normalization 

[Python-Dev] Re: Problems with dict subclassing performance

[Paul Moore]

Has anyone raised this on bugs.python.org? That's the best way to get
something like this looked at, not via a post on Stack Overflow. The
SO posting didn't include a bpo link.
Paul

On Thu, 12 Aug 2021 at 07:33, Marco Sulla  wrote:
>
> No ideas? Excuse me for the up.
>
> On Fri, 6 Aug 2021 at 21:29, Marco Sulla  wrote:
> >
> > I've done an answer on SO about why subclassing `dict` makes the
> > subclass so much slower than `dict`. The answer is interesting:
> >
> > https://stackoverflow.com/questions/59912147/why-does-subclassing-in-python-slow-things-down-so-much
> >
> > What do you think about?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ODHRCDEPOP3AUJJ6IKF2GFLOWJ5BY7TX/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BS3SWNUFGFRHGQFY5KKYJB5C2FHZNOAJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations

[Paul Moore]

On Wed, 11 Aug 2021 at 07:37, Larry Hastings  wrote:
>
[...]
> I think PEP 649 should be considered in the same way.  In my opinion, the 
> important thing is to figure out what semantics we want for the language.

I think this is a very important point. As originally implemented,
annotations allowed attaching arbitrary data to function arguments and
return values. That was a simple language capability, with no
particular nuances, although the intended use case was very much type
annotations. Later, the decision was made that typing would be the
primary and "official" use for annotations - but even at that point,
the semantics were still impartial.

String annotations are changing the fundamental semantics of
annotations (introspection returns the source code string, not a
Python object) and there's even discussion of how this allows the
possibility for annotations to not even conform to standard Python
syntax. Surely, before we start changing the fundamental syntax and
semantics of a language feature for a single (albeit officially
blessed) use case, we should look at what capabilities *in the
abstract* are missing from the current behaviour, and work out how or
if we can implement them. Proposers on python-ideas are routinely
asked to demonstrate how their suggestion affects cases other than
their particular use case. Why shouldn't we hold typing to a similar

PEP 563 states that it's addressing two issues:

* Forward references for type hints
* Module import time cost of having type hints at all

Both of those are real and valid problems that deserve a solution.

PEP 563 is asserting that no solution is possible within the existing
semantics, and it's necessary to significantly break backward
compatibility in terms of semantics, and also opens up the possibility
of annotations not even being normal Python syntax in future (the PEP
says that any such change would need a further PEP, but definitely
does *not* say that the possibility is out of the question). PEP 649,
in my opinion, is exploring the option of addressing these two issues
*within Python's existing syntax and semantics*. How we look at it IMO
depends significantly on how we view the whole question of typing and
what it means for typing to be "optional".

Personally, my experience is that typing is indeed optional for small
or personal projects. But for larger projects, where there's a team of
people involved, it's very hard to argue against typing. Even if
you're not convinced personally that the benefits justify the costs,
it's hard to make that case persuasive, so typing pretty much always
wins.

With that in mind, if we *are* saying (as PEP 563 is, in effect) that
we're at the breaking point where we cannot support the typing use
case with existing Python syntax (note that PEP 563 essentially makes
annotations a new syntax for string literals[1]) then I would far
prefer it if Python gave up on this ambivalent attitude towards
annotations, and made a proper commitment to having a way of
expressing types within the language. That means defining type syntax
as Python *language* syntax and treating type syntax as subject to all
of the compatibility guarantees that the rest of the language adheres
to. It may still mean that we define new, type-specific syntax, but we
can put it in the language definition, and say that it's only valid in
annotations. That's not a showstopper. And it means that we don't open
ourselves up to tool-specific interpretation of core Python syntax.

What that would mean is:

1. What is valid as an annotation would probably need to be specified
once and for all at the language level. That's a compatibility break,
but we'd be making it with our eyes open, having found that the "a
single Python expression" approach isn't sufficient for the typing use
case.
2. The forward reference issue is treated as a *python language*
issue, not just a typing issue. It may be that we decide to only
*solve* it for typing, but we can still look to the wider context for
ideas and inspiration.
3. The idea of having "special syntax" for type expressions would be
dropped - it's not "special syntax" if it's part of the language, any
more than assignment expressions or conditional expressions are
"special syntax".

It's possible that this is something that's already been discussed in
the typing community. But if so, then that's a discussion that should
have involved python-dev, as it's fairly basic to how Python as a
language is evolving.

On the other hand, PEP 647 is saying that we think that typing can
still be supported with existing syntax, and we don't need to (yet)
have that discussion. Although maybe the typing experts have further
plans (the whole "support non-Python syntax" hint suggests that they
might) that mean new syntax is an inevitability - but if so, let's get
that proposal out in the open where it can be discussed, and not use
PEP 563 as a "back door" to avoid that discussion.

[...]
>  Once we figure out what semantics we 

[Python-Dev] Re: Roundup to GitHub Issues migration

[Paul Moore]

On Wed, 23 Jun 2021 at 01:21, Brett Cannon  wrote:
> Regardless, there are no plans to halt what was decided when we accepted PEP 
> 581. Most of the concerns which have been brought up in this thread were 
> already expressed back then (the account merge one I didn't remember, hence 
> why I replied).

I don't have a strong opinion regarding the use of github accounts,
but just to note, this thread was about the open issue in PEP 588, not
about PEP 581 -
https://www.python.org/dev/peps/pep-0588/#a-github-account-should-not-be-a-requirement.

PEP 588 has not been accepted, so it's not necessarily relevant to the
actual migration plan here, but I do think it's reasonable to ask for
some clarification. Either PEP 588 should be rejectected, noting that
the actual implementation plan is being maintained differently, or it
should be updated as an ongoing document as the planning process goes
ahead. I suspect the update on this particular open question might
well be "the problem was considered, and ultimately it was concluded
that requiring a github account was not a showstopper". That may not
please some people (I don't personally care) but that's fine - not
everything has to be unanimous, as long as the SC approves.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UYIGMYAXU25AGLOPEQY2AMWQZ4CCCFZD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.

[Paul Moore]

On Fri, 18 Jun 2021 at 01:57, Guido van Rossum  wrote:
>
> I don’t see how the stable ABI works as a substitute for vendoring Python. A 
> lot of other things can still vary even when the C API remains the same! 
> (E.g. syntax, and stdlib behavior.)

IMO it doesn't. However for certain applications (the sort of thing I
was referring to) - where the user is writing their own scripts and
the embedding API is used merely to expose an interface to the Python
language, dynamically linking to whatever version of Python the user
has installed can be precisely the right thing to do - the user gets
access to the version of the language they expect, the installed
packages they expect to see, etc.

Consider something like the vim integration with Python (which
*doesn't* use the stable ABI, so you have to use a distribution built
for precisely the Python version you have installed - the stable ABI
would be better from the POV of supporting fewer binary builds).
Vendoring is a viable alternative here as well, but as I say, it just
has different trade-offs.

As I said, it's a niche usage, but it has its place. I don't
personally have an opinion on promoting the option of linking to the
system Python - it just suits my use cases.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5LOUHPH6VHNA5KFU4ZE6QC4GKCXSSAHX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.

[Paul Moore]

On Wed, 9 Jun 2021 at 11:36, Inada Naoki  wrote:
> If I am wrong, can we stop keeping stable ABI at Python 3.12?
> Python 4.0 won't come in foreseeable future. Stable ABI blocks Python 
> evolution.

Conversely, the stable ABI allows projects to build cross-version
binary wheels. Not many projects do that yet, but it's definitely
something we'd like to see more of. Needing new binary builds every
version blocks users from testing new versions of Python in advance of
the release. Also, I often use the stable ABI when embedding, so that
I can replace the Python interpreter without needing to recompile my
application and redeploy new binaries everywhere (my use case is
pretty niche, though, so I wouldn't like to claim I represent a
typical user...).

But I do agree that we should either start keeping to the commitments
that we made around the stability of the stable ABI, or we should
abandon it properly and declare it no longer supported. Having
something that sort of works except when we accidentally broke it
doesn't help anyone.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HXOMSUACA5GOH2SFU526R54AJSZGRNXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Paul Moore]

On Thu, 20 May 2021 at 20:06, Ethan Furman  wrote:
>
> On 5/20/21 11:00 AM, Paul Moore wrote:
>
>  > But it nevertheless feels like a bit of an abuse - the original point
>  > of ellipsis was for indexing, and in particular complex slices like
>  > a[1:20:2, ..., 3:5]. That usage is common in numpy, as I understand
>  > it,
>
> Interesting -- do you know what ... means in that context?

In general, it just means a.getitem((slice(1,20,2), Ellipsis,
slice(3,5))), which has no specifically-defined meaning. In numpy, it
means something along the lines of "broadcast along this axis" (I
don't know the numpy terminology very well).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FFKSIHKW2CAQUMBWA6YUDKTGDKNSEQBB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The repr of a sentinel

[Paul Moore]

On Thu, 20 May 2021 at 18:13, Luciano Ramalho  wrote:
>
> I'd like to learn about use cases where `...` (a.k.a. `Ellipsis`) is
> not a good sentinel. It's a pickable singleton testable with `is`,
> readily available, and extremely unlikely to appear in a data stream.
> Its repr is "Ellipsis".

Personally, I'm quite tempted by the idea of using ellipsis. It just
sort of feels reasonable (and in the context `def f(x,
optional_arg=...)` it even looks pretty natural).

But it nevertheless feels like a bit of an abuse - the original point
of ellipsis was for indexing, and in particular complex slices like
a[1:20:2, ..., 3:5]. That usage is common in numpy, as I understand
it, even if it's relatively rare in everyday Python. So while I like
the idea in principle, I'm mildly worried that it's not "the right
thing to do".

I can't put my ambivalence about the idea any more precisely than
this, unfortunately.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZGMZRGHFXQQZZLKBBZKXXAO65TRB6VYX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 659: Specializing Adaptive Interpreter

[Paul Moore]

On Thu, 13 May 2021 at 09:23, Mark Shannon  wrote:
>
> Hi Terry,
>
> On 13/05/2021 5:32 am, Terry Reedy wrote:
> > On 5/12/2021 1:40 PM, Mark Shannon wrote:
> >
> >> This is an informational PEP about a key part of our plan to improve
> >> CPython performance for 3.11 and beyond.
> >
> >> As always, comments and suggestions are welcome.
> >
> > The claim that starts the Motivation section, "Python is widely
> > acknowledged as slow.", has multiple problems. While some people
> > believe, or at least claim to believe "Python is slow", other know that
> > as stated, the latter is false.  Languages do not have a speed, only
> > implementations running code for particular applications have a speed,
> > or a speed relative to equivalent code in another language with a
> > different runtime.
>
> I broadly agree, but CPython is largely synonymous with Python and
> CPython is slower than it could be.
>
> The phrase was not meant to upset anyone.
> How would you rephrase it, bearing in mind that needs to be short?

How about simply "The CPython interpreter, while sufficiently fast for
much use, could be faster"? Along with the following sentence, this
seems to me to state the situation fairly but in a way that motivates
this proposal.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UQFXWM64JORKYJVAGIOV6YJ5OWDTTQRA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Tue, 27 Apr 2021 at 18:01, Brett Cannon  wrote:

> Unfortunately I thought importlib.metadata would have used the module name 
> instead of the metadata details, but in hindsight am guessing that the 
> .dist-info is what it's using to do the lookup and that's based on the 
> package name instead of the project name.
>
> This is a long-standing issue with projects that use project names which 
> differ from their module name, but there's no good way without checking what 
> files a project installed (which is what you're doing below).

That's correct. There is no link from a given import name back to the
PyPI project name. And the metadata for installed packages (defined
here: https://packaging.python.org/specifications/recording-installed-packages/)
is keyed on the PyPI project name, as the OP noted ("beautifulsoup4"
rather than "bs4", "setuptools" rather than "pkg_resources", etc).

>> This is the best I could come up with from reading the docs:
>>
>> import bs4  #<- This is the module we want the version of
>>
>> import importlib
>> import sys
>> from itertools import chain
>> from pathlib import Path
>>
>> loaders = sys.meta_path
>>
>> target_path = Path(bs4.__file__)
>>
>> distros = list(chain(*(finder.find_distributions() for finder in loaders 
>> if hasattr(finder, 'find_distributions'
>> distros_files = chain(*(f for f in (d.files for d in distros)))
>> distro_files = [(d, d.locate_file(f)) for d in distros if d.files for f 
>> in d.files]
>> matching = [d for d, f in distro_files if f == target_path]
>>
>> for match in matching:
>> print("Found Version:", match.version)

The following is a bit simpler. The file.locate() method of a
distribution is undocumented - but I tried to use resolve() on the
path object I got from dist.files, and it appears not to be
implemented (at least in Python 3.9) - PackagePath objects are a
subclass of *Pure* Path objects, not concrete paths :-( TBH, I suspect
the fact that it's undocumented is an oversight, it's clearly
deliberately added.

import importlib.metadata
from pathlib import Path

import pkg_resources
target = Path(pkg_resources.__file__)

for dist in importlib.metadata.distributions():
for file in dist.files:
path = file.locate()
if path == target:
print(f"{dist.metadata['name']}: {dist.version}")
break

To be honest, if anyone were interested in making a PEP from any of
this, having Python modules contain a __distribution_name__ attribute
that links the module back to the PyPI distribution, would probably be
more useful than standardising __version__. But I'm not sufficiently
interested to do anything more than mention that as a possibility.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/66G7A5ZZRYT7H6MRJVELFMC23IDP67CL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: OTish: Define Protocols near consumers [was: Keeping Python a Duck Typed Language.]

[Paul Moore]

On Fri, 23 Apr 2021 at 10:31, Stephen J. Turnbull
 wrote:

> SJIS = 'shift_jisx0213'
>
> *Now* that is a personal protocol, serving the same function of
> telling me "this code deals with a legacy Japanese encoding" and
> implementing it behind the scenes.  But I don't see how that can be
> "defined close to its consumers", which are all over the place,
> including interactive sessions.
>
> What am I missing, and how might that be applied to Python?

What you're missing, I think, is that we're talking about
typing.Protocol - see here:
https://docs.python.org/3/library/typing.html#typing.Protocol

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E4F6APQCEU3CDP22FPVMGYG6NY4LJWON/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 21:40, Adrian Freund  wrote:
> If I understand correctly your concerns with inferring return types for
> inferred protocols are that it might be to restrictive and prevent
> gradual typing. Here are some examples to show how gradual typing would
> still work.

OK, I have no idea what's going on here any more. I have *no* concerns
with inferring the return type. It was you who said that that
inferring would be difficult - the exact quote is "You could
statically type t as Union[Tuple[Literal['version'], int],
Tuple[Literal['name'], str]], but inferring a Protocol for this would
be either very hard or even impossible, especially with even more
complex conditions."

I don't know why you think I have a problem with inferring return
types. All I've ever said is that I thought it might be an interesting
idea if typing an argument as "DuckTyped" could result in type
checkers automatically generated a suitable protocol type, based on
the actual usage of the argument in the function (so that the
programmer doesn't have to explicitly write and maintain a protocol
class in parallel with the code).

> If you have any concrete examples where inferring the return
> type would break gradual typing let me know and I'll have a look at them.

I don't, and I never have. As I say, it seemed to be you who was
claiming that inferring would be too hard.

I don't see much point in continuing this. You seem to be arguing
against points I never made, or maybe I'm completely misunderstanding
you. Either way, we're getting nowhere.

Thanks for taking the time to try to explain, but I think all this has
accomplished is to convince me that there's a "typing mindset" that
embraces a level of strictness that I want nothing to do with. That's
fine, we can agree to differ, but I'm a bit saddened at the thought
that a certain proportion of the information available about typing
might be hard for me to follow because its underlying assumptions are
too different from mine.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KADXCO56OFVLILMWWT5GDRGZERBJUQPB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 15:22, Adrian Freund  wrote:
>
> On April 22, 2021 3:15:27 PM GMT+02:00, Paul Moore  
> wrote:

> >but that's *absolutely* as far as I'd want to go. Note in particular
> >that I don't want to constrain the return value
> The problem is that this isn't enough to have a type safe program. You need 
> to also constrain the return type to make sure the returned value can be 
> safely passed to other functions.

But I don't want a type safe program. At least not in an absolute
sense. All I want is for mypy to catch the occasional error I make
where I pass the wrong parameter. For me, that's the "gradual" in
"gradual typing" - it's not a lifestyle, just a convenience. You seem
to be implying that it's "all or nothing".

> If you don't do this large parts of your codebase will either need explicit 
> annotations or will be unchecked.

That's just not true. (And even if it were true, you're assuming that
I care - I've already said that my goal is much more relaxed than
complete type safety).

I repeat, all I'm proposing is that

def f(x: int): ...
def g(x: str): ...

def main(t: DuckTyped) -> None:
if t[0] == 'version':
f(t[1])
elif t[0] == 'name':
g(t[1])

gets interpreted *exactly* the same as if I'd written

class TType(Protocol):
def __getitem__(self, int): ...

def f(x: int): ...
def g(x: str): ...

def main(t: TType) -> None:
if t[0] == 'version':
f(t[1])
elif t[0] == 'name':
g(t[1])

How can you claim that the second example requires that " large parts
of your codebase will either need explicit annotations or will be
unchecked"? And if the second example doesn't require that, nor does
the first because it's equivalent.

Honestly, this conversation is just reinforcing my suspicion that
people invested in type annotations have a blind spot when it comes to
dealing with people and use cases that don't need to go "all in" with
typing :-(

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CUZNKEA357N3672CCRA7DZ6C4WABN6FK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 13:23, Adrian Freund  wrote:
>
> According to PEP 484 all missing annotations in checked functions should be 
> handled as Any. Any is compatible with all types.

Yep, that's what I understood to be the case.

> I think from a technical standpoint it should be possible to infer protocols 
> for arguments for most functions, but there are some edge cases where this 
> would not be possible, making it impractical to make this the default 
> behavior. Having an annotation to make a type checker infer a protocol would 
> be interesting though.

Absolutely, I see no problem with "use duck typing for this argument"
being opt-in.

> For example:
>
> def f(x: int): ...
> def g(x: str): ...
>
> def main(t):
> if t[0] == 'version':
> f(t[1])
> elif t[0] == 'name':
> g(t[1])
>
>
> You could statically type t as Union[Tuple[Literal['version'], int], 
> Tuple[Literal['name'], str]], but inferring a Protocol for this would be 
> either very hard or even impossible, especially with even more complex 
> conditions.

Yes, but that's inferred static typing which is *not* what I was
proposing. I was suggesting that the checker could easily infer that t
must have a __getitem__ method, and nothing more. So the protocol to
infer is

class TypeOfT(Protocol):
def __getitem__(self, idx): ...

It would be nice to go one step further and infer

class TypeOfT(Protocol):
def __getitem__(self, idx: int): ...

but that's *absolutely* as far as I'd want to go. Note in particular
that I don't want to constrain the return value - we've no way to know
what type it might have in the general case. IMO, inferring anything
else would over-constrain t - there's nothing in the available
information, for example, that says t must be a tuple, or a list, or
that t[3] should have any particular type, or anything like that.

My instinct is that working out that t needs to have a __getitem__
that takes an int is pretty straightforward, as all you have to do is
look at where t is used in the function. Four places, all followed by
[] with a literal integer in the brackets. That's it. I fully
appreciate that writing *code* to do that can be a lot harder than it
looks, but that's an implementation question, not a matter of whether
it's reasonable as a proposal in theory.

This feels like *precisely* where there seems to be a failure of
communication between the static typing and the duck typing worlds. I
have no idea what I said that would make you think that I wanted
anything like that Union type you quoted above. And yet obviously, you
somehow got that message from what I did say.

Anyway, as I said this is just an interesting idea as far as I'm
concerned. I've no actual need for it right now, so I'm happy to leave
it to the mypy developers whether they want to do anything with it.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CU7OES4P3KUCQMD3U5766XVOESL43DRF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 11:21, Paul Moore  wrote:
>
> On Thu, 22 Apr 2021 at 11:06, Chris Angelico  wrote:
> >
> > Someone will likely correct me if this is inaccurate, but my
> > understanding is that that's exactly what you get if you just don't
> > give a type hint. The point of type hints is to give more information
> > to the type checker when it's unable to simply infer from usage and
> > context.
>
> Hmm, I sort of wondered about that as I wrote it. But in which case,
> what's the problem here? My understanding was that people were
> concerned that static typing was somehow in conflict with duck typing,
> but if the static checkers enforce the inferred duck type on untyped
> arguments, then that doesn't seem to be the case. Having said that, I
> thought that untyped arguments were treated as if they had a type of
> "Any", which means "don't type check".

Looks like it doesn't:

> cat .\test.py
def example(f) -> None:
f.close()

import sys
example(12)
example(sys.stdin)
PS 12:00 00:00.009 C:\Work\Scratch\typing
> mypy .\test.py
Success: no issues found in 1 source file

What I was after was something that gave an error on the first call,
but not on the second. Compare this:

> cat .\test.py
from typing import Protocol

class X(Protocol):
def close(self): ...

def example(f: X) -> None:
f.close()

import sys
example(12)
example(sys.stdin)
PS 12:03 00:00.015 C:\Work\Scratch\typing
> mypy .\test.py
test.py:10: error: Argument 1 to "example" has incompatible type
"int"; expected "X"
Found 1 error in 1 file (checked 1 source file)

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54C6G2JLYYD6B37J5KVKPCKSQDCGLRKA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 11:06, Chris Angelico  wrote:
>
> On Thu, Apr 22, 2021 at 7:53 PM Paul Moore  wrote:
> > I wonder whether type checkers could handle a "magic" type (let's call
> > it DuckTyped for now :-)) which basically means "infer a protocol
> > based on usage in this function". So if I do:
> >
> > def my_fn(f: DuckTyped):
> > with f:
> > data = f.read()
> > for line in f:
> > print(line)
> > f.close()
> >
> > then the type checker would automatically build a protocol type like
> > the one I defined above and use that as the type of f? That would make
> > it much easier to include duck typed arguments in function signatures
> > while keeping the benefits of static type checking.
> >
>
> Someone will likely correct me if this is inaccurate, but my
> understanding is that that's exactly what you get if you just don't
> give a type hint. The point of type hints is to give more information
> to the type checker when it's unable to simply infer from usage and
> context.

Hmm, I sort of wondered about that as I wrote it. But in which case,
what's the problem here? My understanding was that people were
concerned that static typing was somehow in conflict with duck typing,
but if the static checkers enforce the inferred duck type on untyped
arguments, then that doesn't seem to be the case. Having said that, I
thought that untyped arguments were treated as if they had a type of
"Any", which means "don't type check".

So I guess the point here is that either the typing
community/documentation isn't doing a very good job of explaining how
duck types and static types work together, or that people who like
duck typed interfaces¹ aren't reading the available documentation in
type checkers explaining how to do that with static typing :-)

Paul

¹ And I include myself in that - maybe I need to go and read the mypy
docs properly rather than just learning what I need by following
examples in existing code...
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IRSXXYHMYCRBBA5HKPNZ7OJDRTEGUAG7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Keeping Python a Duck Typed Language.

[Paul Moore]

On Thu, 22 Apr 2021 at 09:46, Chris Angelico  wrote:
> Maybe in some cases, the "smaller protocols" option is practical, but
> it would need to have a useful name. For instance, if it needs to be
> readable, iterable, closeable, and autocloseable via
> __enter__/__exit__, that's ... uhh a readable, iterable, closeable
> context manager? Not an improvement over "file-like object".

Note: I've not used protocols myself, so this is speculation.

Is the name of the protocol important? Specifically, if I do, in my code

class X(Protocol):
def read(self): ...
def __iter__(self): ...
def close(self): ...
def __enter__(self): ...
def __exit__(self, exc_type, exc_val, exc_tb): ...

def my_fn(fileobj: X) -> None:
# my stuff

would that not work? An argument is checked to see if it conforms with
a protocol by confirming it has the right methods, not by name,
inheritance or registration. And if you want, you can just call X
"FileLike", it's only a local name so it won't clash with whatever
other people (or you, in a different module) want to say is
"file-like". Of course, that's incredibly verbose and messy, and it
would result in a huge proliferation of throw-away protocol classes,
which is probably not a good practice that we'd want to encourage, but
it works.

IMO, the problem isn't that *technically* static typing excludes the
more traditional duck typing, but rather that the design, approach and
as a result the emerging "best practices" are focused around
inheritance based (is that what people mean by "nominal"?) models, to
the point where duck typing feels like an afterthought that you have
to work to include.

I wonder whether type checkers could handle a "magic" type (let's call
it DuckTyped for now :-)) which basically means "infer a protocol
based on usage in this function". So if I do:

def my_fn(f: DuckTyped):
with f:
data = f.read()
for line in f:
print(line)
f.close()

then the type checker would automatically build a protocol type like
the one I defined above and use that as the type of f? That would make
it much easier to include duck typed arguments in function signatures
while keeping the benefits of static type checking.

I will say that at the moment, this doesn't bother me much personally.
On the larger projects where I've used typing, we've been fine with
class-based typing and haven't really needed anything more complex
like protocols. On smaller projects, I just don't use typing at all.
Whether this will change if I decide to introduce typing in more
places, I don't know at the moment. I've also not really used typing
for public APIs, where an over-restrictive type would be more of an
issue.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EQDLTZDXEE7RRFCAVCGLR5OTJOWFVXH5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Thu, 15 Apr 2021 at 21:03, Christopher Barker  wrote:

> I am leaning toward at least SOME PEP about this -- having  __version__ 
> around as a semi-convention with no official recommendation as to if or how 
> it should be used is kind of a mess.

On the other hand, having a PEP that says you "may" have a version
attribute, but mandates nothing, is pretty useless. People who conform
to that convention don't need telling, people who don't have little
reason to change, and people wanting to use the information still have
to cater for it being missing, so they gain nothing.

But a PEP that makes __version__ a requirement fails instantly,
because there's too many packages in the wild that violate it.

Welcome to the world of packaging PEPs :-)

> I'm not sure what to make of all this, though I'm leaning toward better 
> suporting the distiction by asking for __version__ strings in top-level 
> packages -- and maybe making importlib.metadata.version a bit smarter about 
> looking for packages, and not just distributions.

Personally, I'm leaning more and more towards the view that it's
*distributions* that are versioned, and that's an established
practice. Versioning packages as well (except in the very limited
sense that package X installed by distribution Y can expose Y's
version via its API) only adds confusion and makes the situation less,
not more, useful for users.

So I'm now -1 on any PEP to try to formalise the use of __version__.
If people want to discuss the use of a __version__ attribute, they
should consider contributing to the discussions on versions in
https://packaging.python.org/guides/ (where the necessary context is
already present).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ECJ2HHV2EXJ7GOJU7ILLQVVLF54VVD3U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Wed, 14 Apr 2021 at 21:59, David Mertz  wrote:
>
> On Wed, Apr 14, 2021 at 9:12 PM Paul Moore  wrote:
>>
>> If it's not basically equivalent to packaging.version.Version (and
>> based on PEP 440) then we'll be creating a nightmare of confusion,
>> because PEP 440 versions are fundamental to packaging.
>
> Are you suggesting that users should have to install an external module to 
> tell what version of packages they are using?!

No. To tell what version of a package they are using, a string is sufficient.

They only need a version object if they want to do additional
processing (like comparing versions, or checking whether a version
meets a constraint).

Given that the packaging ecosystem already has a canonical version
object (provided by the packaging library), which has been used and
tested extensively in many environments, inventing a different API
seems at best ill-advised. Whether the stdlib needs a version object.
rather than leaving that functionality to a 3rd party library, is the
same question that comes up for *any* functionality that's proposed
for the stdlib, and I have no particular opinion in this case.

>> What's wrong with Version(module.__version__)? And if the __version__
>> attribute isn't a valid version, raise an exception? That's what
>> packaging.version does, and it's worked fine for the packaging
>> ecosystem. Is there a benefit that justifies being different here?
>
> Doesn't that seem really, really painful to use in an interactive shell?  
> This is honestly (a simplified version of) what I tried in trying to learn 
> packaging.version.Version.  It was not a good experience:

It's designed for programmatic use, not interactive use, yes. But
that's sort of my point, why do you want anything more than the bare
string in the REPL? What are you planning on doing with it?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6424E3GOVU463R7SOFQQXCWKV2M3LZRV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Wed, 14 Apr 2021 at 18:04, Eric V. Smith  wrote:
>> Anyway, the question now for me is whether this is worth any more of my time.
>>
>> So:
>> - Is there someone willing to sponsor?
>> - Do folks generally think there's a chance of it being accepted without a 
>> huge debate and expansion of scope.
>>
>> I think that before these can be answered, you need to decide why this needs 
>> to be standardized. I don't see any reason to standardize it unless there's 
>> some programmatic use for these version strings. If it's just the user 
>> poking around on the REPL, then I think the status quo is fine. I've read 
>> the PEP, and the User Stories section talks about user convenience, not 
>> programmatic access.
>
> I also think the distribution version is more useful than any __version__ 
> information in each module/package. I realize this information might not be 
> available, depending on how the code was installed (say, just by copying some 
> files into the right place).

Agreed. The original PEP was very limited, just proposing a string
value for user convenience, with no programmatic interface. Doing
anything more than that is opening up a whole load of complexity and
compatibility (with packaging) that no-one has expressed any need for.

The distribution version (exposed via importlib.metadata) is the
correct thing to use for anything programmatic.

Paul

PS I see Barry plans on rejecting the PEP, which I think is probably
the right decision, given the way this thread has developed.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MQIFSJYQYCY4YBHVECJ7H2WR443EE32W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Wed, 14 Apr 2021 at 18:01, David Mertz  wrote:
>
> On Wed, Apr 14, 2021 at 5:44 PM Christopher Barker  
> wrote:
>>
>> Another possible issue: using Version would require an extra import in many 
>> module initializations -- is that a performance issue that would matter?
>
>
> I like having a `Version` object that is easily importable in the standard 
> library.  I think it should relatively polymorphic.  I.e. it should accept a 
> module or package as an argument, but also should accept a string or a tuple. 
>  Maybe other objects from which one could reasonably extract a version.

If it's not basically equivalent to packaging.version.Version (and
based on PEP 440) then we'll be creating a nightmare of confusion,
because PEP 440 versions are fundamental to packaging.

> In particular, I think initializing this object with a module object should 
> at least look for a .__version__ attribute, and appropriately case either a 
> string (that looks sufficiently version-like) or a tuple.  I think that if it 
> doesn't succeed, it should become some sort of "cannot determine" object that 
> is neither less than nor greater than any other Version object.  In 
> particular, the Vaex example with a dictionary of versions of each component 
> should probably just become this "cannot determine" value (but as an instance 
> of Version).

What's wrong with Version(module.__version__)? And if the __version__
attribute isn't a valid version, raise an exception? That's what
packaging.version does, and it's worked fine for the packaging
ecosystem. Is there a benefit that justifies being different here?

Can I remind people that the packaging community have done a *huge*
amount of work designing and standardising concepts like version
identifiers, version comparisons and constraints, etc. (PEPs 440 and
508). Understanding those standards should really be the basic
starting point for any discussion like this.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U2PUBZZ2AULMULLESO3LZ74MCRV7BQ3G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Wed, 14 Apr 2021 at 13:28, Victor Stinner  wrote:
> There are two main use cases for versions:
>
> * Display them to the user
> * Compare versions to check if one is newer, older or the same
>
> I dislike using strings for comparison. You need to use
> packaging.version for that:
> https://packaging.pypa.io/en/latest/version.html

Agreed, programmatic use should really conform to PEP 440 versioning,
as implemented in packaging.version, if it's to align with existing
packaging standards (which currently focus on *project* versions
rather than package versions, but the distinction is not something we
should be making a big deal of).

> Many C libraries provide the version as a number of as 3 numbers
> (major, minor, micro). In its C API, Python provides all of them:
>
> * PY_VERSION_HEX: single number
> * (PY_MAJOR_VERSION, PY_MINOR_VERSION, PY_MICRO_VERSION,
> PY_RELEASE_LEVEL, PY_RELEASE_SERIAL): as 5 numbers
> * PY_VERSION: string
>
> In my Python projects, I like to provide the version as a tuple which
> can be used directly for comparison: version_a <= version_b. Example:
>
> VERSION = (2, 2, 1)
> __version__ = '.'.join(map(str, VERSION))
>
> The tuple might contain strings like "beta" or "rc", as soon as
> comparison makes sense ;-) Sadly, such tuple is no standardized. Which
> part is the major version? How to format it as a string?
>
> Good luck with trying to standardize that ;-)

Agreed. We should stick to PEP 440 format strings, and not try to
dictate other formats like (named) tuples.

But does the PEP want to cover programmatic use *at all*? It would be
possible to define __version__ as "a human-readable string in PEP 440
format" and note that any use other than for display to a person is
outside the scope of the PEP. That's closer in spirit to the 13-year
old version we currently have, but it does miss an opportunity as we
now do have a standard version class ("standard" in the sense of "a
packaging standard", but not "in the stdlib").

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5LFCLXIPXF3PO6PZ5ZLS7TD2OE4U35OD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Paul Moore]

On Wed, 7 Apr 2021 at 06:15, Stephen J. Turnbull
 wrote:
>
> Greg Ewing writes:
>  > On 7/04/21 5:22 am, Brandt Bucher wrote:
>  > > we might consider updating those templates if the term "Reference
>  > > Implementation" implies a higher standard than "we've put in the
>  > > work to make this happen, and you can try it out here"
>  >
>  > Maybe "prototype implementation" would be better? I think I've used
>  > that term in PEPs before.
>
> That seems to me to correspond well to Brandt's standard as expressed
> above.
>
> To me, "prototype implementation" is somewhere between "proof of
> concept" and "reference implementation", and I welcome the additional
> precision.  The big question is can such terms be used accurately (ie,
> do various people assign similar meanings to them)?
>
> I would define them functionally as
>
> proof of concept
> demonstrates some of the features, especially those that were
> considered "difficult to implement"
>
> prototype implementation
> implements the whole spec, so can be used be developers to
> prototype applications,
>
> reference implementation
> intended to be a complete and accurate implementation of the
> specification

I'm OK with these terms (although I don't actually think you *will*
get sufficient consensus on them to make them unambiguous) but with
one proviso - once the implementation is merged into the CPython
source, I think it should simply be referred to as "the
implementation" and qualifiers should be unnecessary (and should be
considered misleading).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z73B6K5BYFWG6N6FPIO3S3ZCADCYQ7T6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Paul Moore]

On Tue, 6 Apr 2021 at 06:15, Stephen J. Turnbull
 wrote:
>
> Paul Moore writes:
>
>  > It *is* merged and publicly released - it's in the latest 3.10
>  > alpha.
>
> Merged, yes, but in my terminology alphas, betas, and rcs aren't
> "public releases", they're merely "accessible to the public".  (I'm
> happy to adopt your terminology when you're in the conversation, I'm
> just explaining what I meant in my previous post.)

*shrug* It's (in my experience) a continuum - it's not in a release
yet, but it is available via an installer, with a (pre-release)
version number. But I get what you're saying and don't disagree, to be
honest. I see the discrepancy as mostly being because we're trying to
use (imprecise) informal language to pin down precise nuances.

The main point I was making is that it's merged into the CPython
source code at this stage, and available for people to download and
experiment with, which is something I was unclear about.

>  > The fact that the implementation kept getting referred to as the
>  > "reference implementation" confused me into thinking it hadn't been
>  > released yet, and that simply isn't true. Calling it "the
>  > implementation" avoids that confusion, IMO.
>
> The only thing I understand in that paragraph is "that [it hadn't been
> released yet] simply isn't true", which is true enough on your
> definition of "released".  But why does "reference implementation"
> connote "unreleased"?  That seems to be quite different from Mark's
> usage.

In my experience, people developing PEPs will sometimes provide what
gets referred to as a "reference implementation" of the proposal,
which is a PR or equivalent that people can apply and try out if they
want to see how the proposal works in practice. That "reference
implementation" is generally seen as part of the *proposal*, even if
it then becomes the final merged code as well. Once it's released, it
tends to no longer get called the *reference* implementation, as it's
now just the implementation (in CPython) of the feature.

PEP 1 uses this terminology, as well - "Standards Track PEPs consist
of two parts, a design document and a reference implementation" and
"Once a PEP has been accepted, the reference implementation must be
completed. When the reference implementation is complete and
incorporated into the main source code repository, the status will be
changed to "Final"". PEP 635 follows this terminology, with a
"Reference implementation" section linking to the development branch
for the feature.

To put this back into the context of this discussion, when Mark was
referring to the "reference implementation" it made me think that
maybe we were talking about that development branch, and that the code
for the pattern matching PEP hadn't yet been merged to the main
branch, which is why we were still iterating over implementation
details. And that led me to think that they'd better get the
discussion resolved soon, as they risk missing the 3.10 deadline if
things drag on. Which *isn't* the case, and if I'd been following
things more closely I'd have known that, but avoiding the term
"reference implementation" for the merged change would also have
spared my confusion.

> I don't have an objection to your usage, I'd just like us all to
> converge on a set of terms so that Brandt has a compact way of saying
> "as far as I know, for the specification under discussion this
> implementation is completely accurate and folks are welcome to refer
> to the PEP, to the code, or to divergences as seems appropriate to
> them".  I'm not sure if that's exactly what Brandt meant by "reference
> implementation", but that's how I understood it.

Agreed, a common understanding is the main thing here. And as I'm not
an active participant in the discussion, and I now understand the
situation, my views shouldn't have too much weight in deciding what
the best terminology is.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IR6LUUBZ5URVBPFFRHRJJXC33OK3QAYE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Paul Moore]

On Sun, 4 Apr 2021 at 13:49, Stephen J. Turnbull
 wrote:

> Final note: once this is merged and publicly released, it will lose
> its status as reference implementation in the above, strong sense.

It *is* merged and publicly released - it's in the latest 3.10 alpha.
That's really the point I was trying to make with my comment (I'm
steering clear of the "scare quotes" discussion). The fact that the
implementation kept getting referred to as the "reference
implementation" confused me into thinking it hadn't been released yet,
and that simply isn't true. Calling it "the implementation" avoids
that confusion, IMO.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JOFZDHPW5NLI4GGBNBEW5FL36OIAIK3Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Request for comments on final version of PEP 653 (Precise Semantics for Pattern Matching)

[Paul Moore]

On Sun, 4 Apr 2021 at 01:37, Brandt Bucher  wrote:
>
> Mark Shannon said:
> > I was relying on the "reference" implementation, which is also in the PEP.
>
> Can you please stop putting scare quotes around "reference implementation"?

Agreed - apart from the implication Brandt noted, it's also
misleading. The code is in Python 3.10, so the correct term is "the
implementation" (or if you want to be picky, "the CPython
implementation"). To me, the term "reference implementation" implies
"for reference, not yet released".

At this point, we're discussing fixes to an implemented Python 3.10
feature, not tidying up a PEP.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3VFVVRSTVIV4A7MKW65NZRFZQ4CEKUX5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP-376 and PEP-427 interpretation

[Paul Moore]

On Mon, 29 Mar 2021 at 17:40, Theallredman via Python-Dev
 wrote:
> So going back to my actual question SHA224 is disallowed in record files 
> because it's bit length is less then 256?

It doesn't look like it's ever been excluded. The only explicit
exclusions are MD5 and SHA1 as you point out. Do you have a particular
reason to want to use SHA224? Pretty much everyone is using SHA256, as
far as I know.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LVOHVVH72BRRHCTQAGTIK2SZGHRLDEQL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: On the migration from master to main

[Paul Moore]

On Fri, 26 Mar 2021 at 19:54, Mariatta  wrote:
>
>
> The branch has not been renamed. It will be renamed after the release of beta 
> 1.
>
> There is a pending PR on DevGuide on how to change your existing local 
> CPython repository to the new branch:
> https://github.com/python/devguide/pull/662/files#diff-611263234e072dc403733b3dd46d0609cfe44b24d29f05841637b41d093adf0cR157
>
> The details and timeline on the branch renaming can be read here:
>
> https://mail.python.org/archives/list/python-committ...@python.org/message/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/

Thanks, that is the same set of commands I've seen in a few places.
What isn't obvious to me (but may be to the people writing the
documentation) is what happens to branches I have. If I have branch
fix_something that is branched off master from a while ago, do I need
to do anything to it? Or will I just be able to do `git merge main`
once master gets renamed, and it'll just work with the name change
making no difference?

I suspect it *will* just work, in which case all that is needed is a
brief note making that clear. But to many of us, it's not "obvious"
that this is the case.

Paul

PS If I think hard about it, and ponder the git model of blobs and
commits, etc, I can see that no commits change with the renaming, and
the names "master" and "main" are just aliases for particular commits.
So I think I've convinced myself that nothing else is needed. But that
thought process is what I mean by "not obvious" ;-)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EZRUP7UEBAS65QGRSTX7UF5WNGO3DXUB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Merge Request Review Reminder

[Paul Moore]

On Fri, 19 Mar 2021 at 17:26, Hasan Diwan  wrote:
>
> For the side question,  you can always use rebase instead of merge. Let me 
> know if you need further particulars? -- H

There should be no need though, as the Python project has a policy of
squash-merging all PRs.
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DZ3QCEDDZTT5H4IAURXD2YSVXJ4VZZLT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Paul Moore]

On Wed, 17 Mar 2021 at 09:26, Paul Moore  wrote:
> The problem is with the transition - we need to find a way to deal
> with existing `.pth` files, and with people using older version of
> tools (like setuptools and pipx) that write `.pth` files (so we can't
> assume, for example, that Python 3.12 will never see a .pth file using
> the old-style encoding).

Hmm, I just checked and pipx uses UTF-8 when writing .pth files. See
https://github.com/pipxproject/pipx/blob/master/src/pipx/venv.py#L176
(and lol, it was my mistake, I wrote that code -
https://github.com/pipxproject/pipx/pull/168). I'm inclined to report
that as a bug, even though it appears no-one has complained about it.
But that seems counter-productive given the context here.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V5AJK4WZY2JCGZVFI5KY3QD4DYVSSIBB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Paul Moore]

On Wed, 17 Mar 2021 at 08:52, Inada Naoki  wrote:
> On Windows, it must be UTF-8. For example, we use `chcp 65001` in
> `activate.bat` to support unicode path.
> On Unix, raw path is bytestring. So paths can be written as-is. Python
> decode it with fsencoding.

Remember that .pth files contain executable code as well as paths, so
fsencoding is not correct for a .pth file as a whole.

> So I think this is the ideal solution. But this solution requires
> platform-specific code in the site.py.
> I don't think pth files are important enough for this complexity.

.pth files are pretty important in the packaging community. I'd
strongly support making their format and behaviour more precisely
defined.

> Sub-optimal idea is using UTF-8. It is the best encoding for Windows.
> And most Unix systems use UTF-8 too.

+1. IMO, UTF-8 is the only reasonable choice here.

The problem is with the transition - we need to find a way to deal
with existing `.pth` files, and with people using older version of
tools (like setuptools and pipx) that write `.pth` files (so we can't
assume, for example, that Python 3.12 will never see a .pth file using
the old-style encoding).

It's worth noting that using the default encoding is the *correct* way
of writing .pth files at the moment (as that's how site.py reads them
- see https://github.com/python/cpython/blob/master/Lib/site.py#L173)
so this is technically a file format change - tools writing .pth files
will *have* to include version-specific code if they want to support
multiple versions of Python. We need to be very clear about this -
it's not just a case of "tools need to specify the encoding".

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MIZLKDTX2EXEHFKKHO33FRSO7EH62DGW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: pth file encoding

[Paul Moore]

On Wed, 17 Mar 2021 at 08:13, Michał Górny  wrote:
>
> On Wed, 2021-03-17 at 13:55 +0900, Inada Naoki wrote:
> > OK. setuptools doesn't specify encoding at all. So locale-specific
> > encoding is used.
> > We can not fix it in short term.
>
> How about writing paths as bytestrings in the long term?  I think this
> should eliminate the necessity of knowing the correct encoding for
> the filesystem.

If I have a path in my Python program that is "a£b" (a unicode string)
and I want to write it to a .pth file, what encoding should I use to
"write it as a bytestring"? I don't understand what you;re trying to
suggest here.
Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YBE6D37V73OXZYNEW36JO24ZBD7EKAJQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Paul Moore]

Sorry, I keep thinking I've finished and you keep making interesting points :-)

On Wed, 3 Mar 2021 at 17:01, Irit Katriel  wrote:

> Raising an ExceptionGroup is an API change.  If you call APIs that say they 
> will raise ExceptionGroups you need to update your code accordingly. If a 
> library doesn't document that it raises ExceptionGroups and then one of those 
> escapes, then that library has a bug. Just like with any other exception type.

In my experience, libraries don't document what exceptions they raise
very well. You can call that a bug, but it's a fact of life, I'm
afraid. The problem here isn't so much that the library code now
raises an exception that it used not to raise, but rather that *the
user hitting Ctrl-C* can now result in a different exception surfacing
in my code than it used to. Libraries don't re-wrap KeyboardInterrupt,
as you pointed out in a previous response, so I can currently write
code that traps KeyboardInterrupt, safe in the knowledge that by doing
so I'll handle that user action properly. But with PEP 654, libraries
might well (indeed, some libraries almost certainly will) start
wrapping KeyboardInterrupt in an exception group. That's a backward
incompatible change from the perspective of my code's interaction with
the user, and I need to re-code my application to deal with it (and
worse still, writing that new code in a way that is portable between
versions is not particularly straightforward).

> For older Pythons you would have to do something like
>
> except KeyboardInterrupt:
>...
> except BaseExceptionGroup:  # some stub type in old versions
># inspect the contents
># if there's a KeyboardInterrupt do what you need to do
># reraise the rest

I'd be inclined to suggest that a complete version of this should be
included in the "Backward compatibility" part of the PEP, as I
honestly don't really know how I'd write that without doing more
research. But such an example would make the KeyboardInterrupt case
seem more important than it is. Maybe if it's framed as "how to write
calling code that's compatible with older versions of Python but still
able to handle called code potentially raising exceptions that you
need to trap as part of a group", that would be a useful general
example.

Or maybe it's not actually something that will be needed that often.
I'm not sure - I'm trying to think in terms of pip, where we can't use
new features in our own code until we drop support for older versions,
but we might potentially rely on a library that uses exception
grouping internally on versions where it's available (and that code
lets those exception groups escape). It feels like a stretch to claim
this is particularly likely, but conversely it's something I can
easily imagine *could* happen...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ITBRA7QAU64BN3XKBYV6G35NXKK2ESXX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Paul Moore]

On Wed, 3 Mar 2021 at 14:59, Irit Katriel  wrote:
>> 2. Given the above, why even have a means of grouping BaseExceptions
>> at all? Why not just have ExceptionGroup that can only catch instances
>> of Exception?
>
> Because the IGotInterrupted alternative involves wrapping a BaseException by 
> an Exception, which is not something we should push people into doing (it's 
> not that different from allowing ExceptionGroup to wrap BaseExceptions 
> directly).

That's a fair point.

> What's the harm/complication in offering a BaseExceptionGroup(BaseException) 
> in addition to ExceptionGroup(BaseExceptionGroup, Exception)?

Similar to the argument for "except Exception". Applications that trap
KeyboardInterrupt so that they can exit cleanly without an ugly
traceback will no longer trap *all* keyboard interrupts, as they could
miss grouped ones.

If we accept that grouped exceptions should never escape out of a
well-defined context, then this wouldn't be such an issue. But there's
nothing in the PEP that enforces that, and there *is* code that needs
to be prepared for "any sort of result". It's the except Exception
argument again.

So code that wants to exit cleanly in the face of Ctrl-C will need to
be rewritten from:

try:
main()
except KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)

to:

try:
try:
main()
except KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)
except *KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)

Did I miss an easier way of writing this code? And worse, how would I
write it so that it was portable between Python 3.9 and later versions
(which is a common requirement for library code - admittedly library
code wouldn't normally be doing this sort of top-level trap, but it
could just as easily be "catch Ctrl-C and do a bit of tidy-up and
re-raise").

> I think the only reason you're comfortable with having to select between the 
> exceptions that were raised and discard some of them is because that's where 
> we are today. The PEP lists several standard library and other APIs that 
> discard errors because they need to pick one. That's what we're trying to fix.

Maybe. But I'm not looking at it as being "comfortable" with the
current situation, but rather as "I don't use any of these new
features, why am I having to change my code to accommodate stuff I
don't use?" If I own the full stack, that's not an issue, but
frameworks and libraries typically have to interact with other users'
code, and there the contract has changed from "do what you like in
your code and I'll cope" to "do what you like in your code as long as
you don't let an exception group escape, and I'll cope"... And I have
to change *my* code to get the old contract back.

But it's a small point in the wider scheme of things, and I'm not
going to labour the point any more. Thanks for listening and taking
the time to reply.

Paul

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WSUEOGDCBBOZ7PCQGUCXKIZEZ7RK34LK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Paul Moore]

On Wed, 3 Mar 2021 at 12:37, Irit Katriel  wrote:
> This is covered the PEP, but TL;DR:
> If we could make "except KeyboardInterrupt" catch 
> BaseExceptionGroup([KeyboardInterrupt]) in a reasonably backwards compatible 
> way then we wouldn't need except*.
[...]
> As you noted, no library is under any obligation to wrap KeyboardInterrupts 
> into the exception groups it raises. You may decide it's a bad idea and not 
> do it.  What we are discussing here is what the language should make 
> possible. We agree that wrapping a BaseException by an Exception is something 
> we should definitely block. When it's wrapping a BaseException by another, 
> new BaseException type, in my view that's ok. You may have a bug where you 
> don't catch an exception you want to catch, because you are using a new API 
> incorrectly. But you won't have bugs where you swallow an exception that you 
> didn't swallow before.

Thanks for the explanation. I understand your point here, and I see
what you're saying. But I have a couple of questions still:

1. Having now read the PEP, I don't actually see a use case for
grouping BaseExceptions. Why would anyone catch KeyboardInterrupt or
SystemExit and wrap them in a BaseExceptionGroup anyway? It seems to
me that the right thing to do, even in async or parallel code, is to
just propogate the KeyboardInterrupt/SystemExit up to the main
program. Losing a ValueError that happened at the exact same time as
the user pressed Ctrl-C seems like it's not going to be a problem in
practice...
2. Given the above, why even have a means of grouping BaseExceptions
at all? Why not just have ExceptionGroup that can only catch instances
of Exception?

If there really *was* a low-level case where some code absolutely had
to (temporarily) group KeyboardInterrupt, for example, it could be
temporarily wrapped:

class IGotInterrupted(Exception):
def __init__(self, exc):
self.exc = exc

def might_be_interrupted():
try:
critical_stuff()
except KeyboardInterrupt as exc:
raise IGotInterrupted(exc)

def funky_parallel_stuff():
try:
do_in_parallel(might_be_interrupted)
except *IGotInterrupted as e:
# Please excuse the fact that I don't know the
# correct way to re-raise here. Might need raise from.
raise e.exc
# Do we also need to consider an *ungrouped* IGotInterrupted?
# That's for the library to decide, and is why not letting it
escape is a good thing...

That would be appropriate for really low-level code, and it would be a
fairly obvious anti-pattern for an exception class like
IGotInterrupted to "escape" so that user code would ever see it.

I guess the argument for grouping BaseExceptions is "why make it
necessary to do complicated wrapping like this?" To which my answer is
"because it should almost never be something you do, and so why add
support for it when it can be done *without* needing explicit support
in the PEP?" And by not having stdlib support for wrapping
BaseExceptions, we're signalling that we don't think people should be
doing it casually...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5LVHBCIWOLYVUYAMFYIZAPEDV7JUVBYS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Paul Moore]

On Tue, 2 Mar 2021 at 21:46, Irit Katriel via Python-Dev
 wrote:
> As an aside - I found it interesting that the option to wrap BaseException 
> instances by an Exception, which came up a couple of times in this thread, 
> didn't generate any anxiety.

Probably because it wasn't clear that was ever being proposed... (or
at least the implication wasn't obvious - presumably this is somehow
related to BaseExceptions being accumulated in ExceptionGroups?) :-(

I would consider it essential that if someone hits Ctrl-C and that
generates a KeyboardInterrupt, then:

1. That KeyboardInterrupt will *not* get caught by exception handlers
only interested in Exception instances
2. That KeyboardInterrupt *will* get caught by any handler that does
an explicit `except KeyboardInterrupt` or an `except BaseException`.

To me, that's pretty much essential to correct Ctrl-C handling in any
app (never ignore a user's Ctrl-C and always exit cleanly if one is
raised).

That might mean that BaseException instances shouldn't be "groupable",
but I don't want to comment on that until I've properly read the PEP
(I've skimmed it now, but only superficially). At a minimum, I'd
consider it a bug for library code to manually wrap a
KeyboardInterrupt in an exception group (just like code that catches
KeyboardInterrupt and re-raises it as a ValueError would be today).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5SPIFHGWASEGKBITWPDDRXOU33NLG46/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Which thing is "Development Mode"

[Paul Moore]

I'm somewhat baffled by your comment here. "Just because something is
bad or even often bad, is no reason to let it stay that way or to not
try and make things better." I have no idea what gives you the
impression that I think that way.

*Shrug* I guess I don't understand what you expect here. But if you
want to champion a change, I'm not going to stop you.

My comment on the packaging terminology stands, though - you'd be
better referring to that as "editable mode" as that is common usage,
*and* it doesn't clash with the cpython option.
Paul

On Tue, 2 Mar 2021 at 16:12, Coyot Linden (Glenn Glazer)
 wrote:
>
> On 3/1/21 08:49, Paul Moore wrote:
>
> On Mon, 1 Mar 2021 at 16:25, Coyot Linden (Glenn Glazer)
>  wrote:
>
> This is one of those cases, where those of us who have been around
> Python for awhile can perhaps infer from context which his meant, but
> the phrase "put your Python project into development mode" is ambiguous
> in a way that is different from your example.
>
> The commonly used packaging terminology is actually "editable mode"
> (at least in the sense that that's what most people seem to call it,
> so knowing and using that term will get better results from Google
> ;-)). If you feel inclined to raise a PR against the pip documentation
> to prefer that term, it can be discussed over there (but note that the
> underlying setuptools command is `setup.py develop` and that's not
> likely to change, as it's hard-coded in a number of applications,
> including pip).
>
> But ultimately, I think the reality is that there *are* terminology
> confusions all over the place in computing, much like in any other
> discipline (I'm currently teaching someone music, and the number of
> terms that get re-used in different contexts there is even worse than
> computing!) So I think that ultimately, beginners simply need to get
> used to understanding that terms can be context-dependent. It's not
> easy, and as teachers we can do what we can to make things clearer,
> but we can't ever completely make such problems go away.
>
> Paul
>
>
> I think we have fundamentally different worldviews, Paul. Just because 
> something is bad or even often bad, is no reason to let it stay that way or 
> to not try and make things better. Engineering almost never makes any problem 
> completely go away, the perfect can't be the enemy of the good.
>
> Personally, since it is newer and not stuck with the problems you mention, I 
> would prefer to change the name of the new Python runtime option than mess 
> with the setuptools mess.
>
> Best,
>
> coyot
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VRLGKI5J6FFX7KFLFOCM2GI473CM5EFV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Which thing is "Development Mode"

[Paul Moore]

On Mon, 1 Mar 2021 at 16:25, Coyot Linden (Glenn Glazer)
 wrote:
> This is one of those cases, where those of us who have been around
> Python for awhile can perhaps infer from context which his meant, but
> the phrase "put your Python project into development mode" is ambiguous
> in a way that is different from your example.

The commonly used packaging terminology is actually "editable mode"
(at least in the sense that that's what most people seem to call it,
so knowing and using that term will get better results from Google
;-)). If you feel inclined to raise a PR against the pip documentation
to prefer that term, it can be discussed over there (but note that the
underlying setuptools command is `setup.py develop` and that's not
likely to change, as it's hard-coded in a number of applications,
including pip).

But ultimately, I think the reality is that there *are* terminology
confusions all over the place in computing, much like in any other
discipline (I'm currently teaching someone music, and the number of
terms that get re-used in different contexts there is even worse than
computing!) So I think that ultimately, beginners simply need to get
used to understanding that terms can be context-dependent. It's not
easy, and as teachers we can do what we can to make things clearer,
but we can't ever completely make such problems go away.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RKT7DFMENJTZAQDTLAEGQT6O23I3ZFIK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Sat, 27 Feb 2021 at 13:03, Oscar Benjamin  wrote:
> More recently I've been exploring Julia. You could compare these pages:
>
> https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter
> https://docs.julialang.org/en/v1/manual/getting-started/
>
> When installing julia you select the installer for your OS and there
> are some instructions for setting up PATH (although I didn't need to
> do that manually on OSX). After setup the instruction is that you type
> "julia" in the terminal and then Ctrl-D to exit. There are no caveats
> or mentions of different operating systems or words like "if" and
> "usually". The setup process is platform specific but then the usage
> instructions are not.

Sorry, I couldn't resist replying to this.

I went to the Julia page. That linked me to the download page. Which
offered me multiple versions. Which should I take? Stable, LTS or
"upcoming"? As a beginner without the experience I have hitting stuff
like this, how would I know? Let's assume "stable". OK, download and
run an installer is easy enough for me on Windows, but let's say I'm
doing this on Linux, I get the choice of "Generic Linux on x86" or
"Generic Linux on ARM". Huh? I'm on Ubuntu, not "Generic", so I'm
confused. Then I have "64-bit (GPG)" or "64-bit (musl) (GPG)" Or AArch
if I chose "ARM". How the heck would I know. Any of those get me a
.tar.gz file, which I don't know how to use...

Going to the Linux help, I get told to use wget, not the file I just
downloaded. And then I unpack it, and now I'm into setting PATH to
/tmp/julia-something/bin (because I did this in /tmp, as I have no
idea where I want to put this long term).

OK, my linux thought experiment isn't going well. Let's go back to
Windows, where I was lucky enough to pick "installer" rather than
"portable", so I avoided a world of pain about where to put the
package, just like I hit on Linux. The installer puts Julia in my
personal apps directory, just like Python does. I'm going to do
something "advanced" here ;-) and deselect "desktop shortcut" because
I hate clutter on my desktop. OK, it's complete, let's select "run
julia". That got me a prompt. Cool. Start menu item does the same.
Also cool.

BUT... The python installer on windows is *exactly the same*. Installs
to user's area, start menu entry that runs the REPL. So that's no
worse.

Suppose I go to a command prompt on my own and try to run Julia. Open
a terminal, "julia". No good, I need to set PATH. Hmm, wasn't there
something in the help on the Julia website about that? Off we go to
check. Weren't we here before when we were using Python? And the
instructions on the Julia website are about as complex as I'd expect,
and much the same as for Python. So still no worse for Python. But the
Python installer has "add Python to PATH" that does this stuff for you
(not selected by default, because of reasons that you may or may not
agree with, but at least it's *there*). And there's the py launcher,
but let's ignore that, as it works round some complexity at the cost
of some different complexity, so it's a distraction at this stage.

Honestly, I think Julia is overall slightly *worse* than Python (at
least on Windows). But it's almost identical in practice.

The one thing that *is* substantially worse for Python, is the
circumlocutions needed in the documentation to say how to run Python.
But that's 100% down to us not being willing to say "just type the
command python". And the reason for *that* is mostly historical,
related to the Python 3 transition and what happened on Linux over the
python/python2/python3 commands, and to a lesser extent the
introduction of the launcher on Windows ("lesser", because using the
launcher on the command line wasn't recommended until some time after
its introduction, at which point the python2/python3 split was
established)...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7AVAVWMGNRDLAEQHDLWWTPHMLJG4V7RD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Paul Moore]

On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin  wrote:
>
> On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett  wrote:
> >
> > I think his point is that most of his students (economics or business, 
> > rather than comp sci) will never need to use Perl or C or Java.  Python is 
> > friendly enough to be useful, but this is still a major pain point.
>
> Thanks Jim, that is the situation and yes they are not CS students.

I've been in this situation with people who are professional
developers (but in environments where "development" is done in a
self-contained application IDE). I completely understand and
sympathise with the idea that suddenly having to learn command line
skills when all you'd signed up for was to learn Python is both
daunting and off-putting.

> The other point though is that it doesn't need to be like this. If the
> issue was just installing Python and then setting up your PATH then
> that's manageable. The problem is that even after doing those things
> there isn't a "one way" to invoke Python from the command line. All
> possible invocations (python, python3, py, ...) will fail for some
> users. That's a problem for beginners but it's also a problem in any
> situation where you want to write a command line that should run on
> multiple platforms (e.g. in a Makefile or some other kind of script).

This is 100% true. (At least the part where you say that "it's a
problem". I'm not sure if "it doesn't need to be like this" is true -
can you point to a language where the command line "just works" the
way you suggest?)

> I see that the official Python tutorial now suggests typing
> "python3.9" [1]. Is that what is recommended now? Obviously that would
> fail for someone who had installed 3.8.
>
> It would be great if Python could converge around a single way for
> users to invoke the Python that they have installed and would want to
> use. Python 2.x is beginning to fade away so python vs python3 makes
> less sense now but maybe standardising on py for all platforms and
> installs would be better (since py also has features for selecting
> different versions and is not currently used for any "system python").

It *would* be great. I don't think there's anyone who disagrees with
this point. But just thinking "it would be great" isn't of much
practical use. People have been trying to fix this issue for many
years (quite apart from all of the core Python debates, it's an
endless, frustrating topic in the packaging community - I can't tell
you how many times we've tried to come up with a good way to tell
people how to invoke pip).

In my view, for people learning Python in a context where command line
skills are *not* the norm, it would be ideal to focus on one (or more)
"integrated environment", in much the same way as no-one learns
command-line R, they all use R Studio. Unfortunately, the Python
community isn't good at getting behind a single solution like this, so
we have IDLE, Jupyter, Spyder, and probably a host of others.

The command is *hard*. People used to it tend to forget that. I know
people who are good developers (in an IDE) who struggle with how to
specify options to commands, and how to quote arguments. (Not
surprising, as both of those depend on what OS and what shell you're
using, as well as being entirely learned information - there's no
"intuitive" reason why options start with a dash, or you quote with
double quotes not single quotes in Windows). I don't think we should
be pointing beginners at the command line *at all*. But as a
community, we don't really have any better option. (Individual
trainers or organisations can make their own choices, but that's not
easy in the absence of a community consensus).

Paul

PS Having a standard "integrated environment" that people learn in has
its own problems, of course, as we risk training a new generation of
Python users who don't know how to use their Python code outside of
that IDE, and who therefore can't work with git or automated tests,
etc, etc... I'm looking at Java and the "if my IDE doesn't do it, I'm
lost" mindset here. I don't know if R has fallen into a similar trap
with R Studio.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3FEHMUITOB4CW3TFQKFBAS2AEEXJEUXE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Paul Moore]

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett  wrote:
>
> Whenever I've used except Exception or stronger, it was a sanitary barrier 
> around code that might well do unpredictable or even stupid things.  Adding a 
> new kind of exception that I hadn't predicted -- including ExceptionGroup -- 
> would certainly fit this description, and I want my driver loop to do what I 
> told it.  (Probably log an unexpected exception, and continue with the next 
> record.  I honestly don't even want a page, let alone a crash, because data 
> from outside that barrier ... is often bad, and almost never in ways the 
> on-call person can safely fix.  And if they don't have time to find it in the 
> logs, then it isn't a priority that week.)

This is my biggest concern. Disclaimer: I've yet to read the PEP,
because async makes my head hurt, but I am aware of some of the
background with Trio. Please take this as the perspective of someone
thinking "I don't use async/await in my code, can I assume this
doesn't affect me?"

For me, the biggest requirement I would have is that if I have code like:

def safe_fn():
try:
do_something_funky()
return True
except Exception:
print("Well, that was a shame...")
return False

then I am intending to guarantee that calling safe_fn() will never
raise an exception. Obviously, the example is a toy (in reality, I'd
log the error, skip a record, whatever) but that's not the point here
- the point is that I consider it reasonable to expect `except
Exception` to be a hard barrier that allows me to be sure I've covered
everything.

My worry is that if ExceptionGroup exceptions are *not* trapped by
`except Exception`, then code like this is no longer safe. And by
making ExceptionGroup a BaseException subclass, that's what will
happen.

Ultimately, there's a genuine use case for code that says "whatever
happens in the code I call, this is my last line of defense and I want
to be as near to 100% sure as I can be that I regain control at this
point". At the moment, that is spelled `except Exception`. If you're
proposing to change that, even if it's just to require that it be
spelled differently, you're going to break quite a lot of code - and
worse, the code you're breaking will be code that has taken care to
ensure it's written safely, so you're penalising people who *tried to
get stuff right*, which IMO is the worst type of breakage.

I will at some point read the PEP fully, to better understand the
async side of the story, but for now please consider this as the
perspective of someone who doesn't expect to care about async, and
therefore wants to feel safe in assuming this PEP won't affect them.

Paul

PS If you're thinking "but if you were using async, you'd know about
it", consider that I could be using a PyPI library that, for example,
returned stock values. That library switches to using async, and has a
bug that lets an exception group escape. My `except Exception` code is
designed *precisely* to protect me against such bugs, without me
needing to know anything about how the library works...
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SKU4RIRPRRGESKRMP55ZPZNYL4XWFFUR/
Code of Conduct: http://python.org/psf/codeofconduct/