[Python-Dev] multiple inheritance and `__str__`

[Ethan Furman]

I ran into an issue today with `str()` not behaving as I thought it should.

Given the following test script, what should happen?

-- 8< --

class Blah(object):
def __str__(self):
return 'blah'

class Huh(int, Blah):
pass

class Hah(Blah, int):
pass

huh = Huh(7)
hah = Hah(13)

print(huh)
print(hah)

-- 8< --

I thought I would get:

7
blah

and indeed, that is what I got for Pythons 2.7 - 3.7.  However, starting with 
Python 3.8 I started getting:

blah
blah

To say the least, I was surprised.

Some searching turned up issue 36793: "Do not define unneeded __str__ equal to 
__repr__" .

As can be seen, `__str__` is needed for inheritance to work properly.  Thoughts 
on reverting that patch?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GLHE6CJ3IEHPUTA36YWOF5FSDLYNIOCV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Christopher Barker]

On Tue, Nov 30, 2021 at 3:31 PM Guido van Rossum  wrote:

>
>> There is some discussion of the numeric tower in PEP 484 but the PEP says
> you should just use 'int', 'float' and be happy.
>

Thanks -- I didn't think to look there. And this: "when an argument is
annotated as having type float, an argument of type int is acceptable"
makes it pretty workable.

I do wonder what happens  with oddball numbers, like, say numpy.float32.
But maybe that's Ok, as for though most part one will be dealing with
arrays of them -- the scalars are fairly rare in real code.

however, IIRC, __index__ was introduced at least partially so that numpy
integer types could be used as indexes. Which would mean SupportsIndex for
a type that's going to be used as an index, yes?


> solutions is just terrible (too much typing for too little benefit) and
> honestly in many cases the runtime is not very good at these either (e.g.
> Decimal refuses to play). In practice it is just fiction that you can write
> your own numeric type.
>

Yes, Decimal and Fraction are pretty special purpose, really.


> Regarding Iterable[int] vs. Sequence[int], there are subtle semantic
> differences so it depends, but these have much better support and *are*
> encouraged. (What helps is that List is invariant but Sequence is
> covariant, and people often want covariance, so there is a real incentive
> to use Sequence, Iterable etc.)
>
>
>> But I'd really love to see the community as a whole start with more
>> generic types in examples, and be clear that the concrete types should be
>> used only when necessary -- even the typing module docs use a lot of
>> concrete types in the examples.
>>
>
> Maybe you are speaking with a lot of enthusiasm but not a lot of
> experience?
>

well, yes, for sure.


> Generics aren't always better, and in many cases they just aren't worth
> the effort to get them right
>

I suspect that depends a bit on the use case. For something internal to one
system, probably not worth the effort at all. But for a general purpose
library, I'm (perhaps needlessly) concerned about types getting locked down
more than they need to be, just because its easier, and "it works for me"


> (just like not every bit of code you write needs to be published on PyPI).
>

Someone should tell that to all the folks publishing 0.0.0.1 version of
packages :-)


> I also have a feeling that most frameworks that use *runtime*
> introspection of types strongly prefer concrete types, i.e. list[int]
> rather than Sequence[T].
>

This is getting into what it means to use Types at runtime -- if it's
runtime type checking, then more abstract types might be fine. But if it's
using the type itself (like my use case), then absolutely -- my system will
only work with real concrete type objects. Though that may not play that
well with type checkers :-(


> It would be nice if someone did some work and collected a list of
> tutorials about type annotations that exist (especially the ones that are
> discoverable with a simple Bing query) and ranked them by quality.
>

yes, it would.


> We should definitely push back on zealous new converts to typing who
> insist that everything should be annotated.
>

well, we got folks wanting to change PEP 8 becuase they don't want their
linter to complain -- so it will be a battle.


> (Have you run into VS Code yet? It gets tremendous value from typing
> stubs, in the form of improved auto-complete and hover-doc functionality.)
>

Some of the folks on my team use it -- I've been meaning to check it out --
one more reason now.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PTZBYWYU66JVOWVHKISKDBMQZDLWEK43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Steve Dower]

On 12/1/2021 12:47 AM, Oscar Benjamin wrote:

On Tue, 30 Nov 2021 at 23:37, Guido van Rossum  wrote:


We should definitely push back on zealous new converts to typing who insist 
that everything should be annotated. But we should also recognize that even in 
their current, far from perfect state, type annotations can provide a lot of 
value, when used right. (Have you run into VS Code yet? It gets tremendous 
value from typing stubs, in the form of improved auto-complete and hover-doc 
functionality.)


I might be misunderstanding but if "hover-doc" is the same as
"mouse-over" then it apparently needs many gigabytes of memory and a
lot of CPU time when you put the mouse over a sympy function that you
are using in your code. It has been suggested that SymPy should fix
this by adding hints like Any but I don't see the point of that: SymPy
has plenty of bugs but this particular bug is in VS Code.


For the record, this is the *old* support that has since been replaced 
(and was based on full program analysis in the Python of ~2010 that had 
no annotations at all, so always struggled with programs that could not 
be thoroughly inferred and particularly with those that generated 
runtime types - Sympy was a special case in it since about 2012 to avoid 
those issues, but that may have been lost when it was migrated from 
Visual Studio to VS Code).


The latest versions of VS Code have a completely new system, that does 
rely very heavily on programs and libraries being annotated, in exchange 
for being much lighter on memory use and preprocessing. But it should 
stay out of your way on unannotated code if you haven't enabled its more 
strict modes.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LHNW4CDJVJKNWTCJSTLVMBOBFUYMSWGZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Guido van Rossum]

On Tue, Nov 30, 2021 at 4:48 PM Oscar Benjamin 
wrote:

> On Tue, 30 Nov 2021 at 23:37, Guido van Rossum  wrote:
> >
> > We should definitely push back on zealous new converts to typing who
> insist that everything should be annotated. But we should also recognize
> that even in their current, far from perfect state, type annotations can
> provide a lot of value, when used right. (Have you run into VS Code yet? It
> gets tremendous value from typing stubs, in the form of improved
> auto-complete and hover-doc functionality.)
>
> I might be misunderstanding but if "hover-doc" is the same as
> "mouse-over" then it apparently needs many gigabytes of memory and a
> lot of CPU time when you put the mouse over a sympy function that you
> are using in your code. It has been suggested that SymPy should fix
> this by adding hints like Any but I don't see the point of that: SymPy
> has plenty of bugs but this particular bug is in VS Code.
>

(Yes, I meant mouse-over. I had a senior moment. :-) I cannot help you much
here, there's a tracker and generally speaking the team does pay attention
to tracker issues, so I recommend starting a discussion there. Feel free to
CC me on the issue if you decide to create one.


> I can see the potential value that type hints can bring for an editor
> like VS Code but is the editor also pushing an agenda that all code
> should be *explicitly* typed? Does VS Code pressurise people into
> using type hints in their own code? I presume it does because I see
> the effect in my students (although I might be misattributing this to
> VS Code rather than something else).
>

There are two modes in VS Code, a lenient and a strict mode. The strict
mode does insist on typing, but you have to opt in to it first. Once you've
opted in, it does indeed encourage you to type everything -- but that's
pretty much the meaning of strict type checking (mypy has a --strict flag
that also requires this).

That said, your students probably copied some configuration that turns on
strict mode from someone else who is thereby pushing this agenda
unwittingly. Sadly most people forget where they got their initial
configuration (I know I forget it as soon as it works :-). And the strict
mode also provides useful error messages in certain cases.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3M6UP6TNKMVJTK4JWL75S6URUERFEJIE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Oscar Benjamin]

On Tue, 30 Nov 2021 at 23:37, Guido van Rossum  wrote:
>
> We should definitely push back on zealous new converts to typing who insist 
> that everything should be annotated. But we should also recognize that even 
> in their current, far from perfect state, type annotations can provide a lot 
> of value, when used right. (Have you run into VS Code yet? It gets tremendous 
> value from typing stubs, in the form of improved auto-complete and hover-doc 
> functionality.)

I might be misunderstanding but if "hover-doc" is the same as
"mouse-over" then it apparently needs many gigabytes of memory and a
lot of CPU time when you put the mouse over a sympy function that you
are using in your code. It has been suggested that SymPy should fix
this by adding hints like Any but I don't see the point of that: SymPy
has plenty of bugs but this particular bug is in VS Code.

I can see the potential value that type hints can bring for an editor
like VS Code but is the editor also pushing an agenda that all code
should be *explicitly* typed? Does VS Code pressurise people into
using type hints in their own code? I presume it does because I see
the effect in my students (although I might be misattributing this to
VS Code rather than something else).

--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C2EEXBT6BEWGOUHBDZ4ILWN6MQZUGQAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Rob Cliffe via Python-Dev]

Heh. We could update PEP 8 to ban type annotations, then watch as the
people who over-zealously apply PEP 8 to everything AND
over-zealously
insist on adding type annotations to everything have their heads
explode.


-- 
Steve



I love it!
"Surtout, pas trop de zèle".  [Charles-Maurice de Talleyrand]
Rob Cliffe___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EWPCRCFOPXXL2QLYMH7QADKXE7Q7PM5P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Guido van Rossum]

On Tue, Nov 30, 2021 at 2:52 PM Christopher Barker 
wrote:

> Another concern I have is the over specification of types.
>
> I have seen many examples of, e.g.
>
> func(x: int, y: float, stuff: List(int]):
>
> but very few of:
>
> func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]):
>
> (or even Iterable[int])
>
> Is that even the right thing to do to get generic number types? How do I
> specify a type that could be any number type (float, int, Fraction,
> Decimal, numpy.float32) ... I just spent a few minutes perusing the typing
> module and MyPy docs, and didn't find a discussion of that.
>

There is some discussion of the numeric tower in PEP 484 but the PEP says
you should just use 'int', 'float' and be happy. *If* we were to change our
mind on that we would let you spell it using numbers.Integer, numbers.Real
etc., not SupportsInt, SupportsFloat. But the usability of any of those
generic solutions is just terrible (too much typing for too little benefit)
and honestly in many cases the runtime is not very good at these either
(e.g. Decimal refuses to play). In practice it is just fiction that you can
write your own numeric type.

Regarding Iterable[int] vs. Sequence[int], there are subtle semantic
differences so it depends, but these have much better support and *are*
encouraged. (What helps is that List is invariant but Sequence is
covariant, and people often want covariance, so there is a real incentive
to use Sequence, Iterable etc.)


> But I'd really love to see the community as a whole start with more
> generic types in examples, and be clear that the concrete types should be
> used only when necessary -- even the typing module docs use a lot of
> concrete types in the examples.
>

Maybe you are speaking with a lot of enthusiasm but not a lot of
experience? Generics aren't always better, and in many cases they just
aren't worth the effort to get them right (just like not every bit of code
you write needs to be published on PyPI).

I also have a feeling that most frameworks that use *runtime* introspection
of types strongly prefer concrete types, i.e. list[int] rather than
Sequence[T].


> I think this is an issue for two reasons:
>
> 1) Folks learn from examples more than instruction -- people are very
> likely to follow the first example they see that seems to do the job.
>

It would be nice if someone did some work and collected a list of tutorials
about type annotations that exist (especially the ones that are
discoverable with a simple Bing query) and ranked them by quality.


> 2) as mentioned in this thread, library authors are being asked to type
> hint their libraries. I think it will be all too common for that type
> hinting to be more specific than required. For the most part, that won't
> cause any real issues right away, but as soon as someone uses that lib in a
> large project that enforces static type checking, it's could get ugly.
>

That's just the tip of the iceberg. In practice, there are quite a few
reasons why the first few iterations of type hints for many popular
packages fall short of expectations. Often the issue isn't just
sufficiently wide types (like you speculate here) but missing types (for
example, stubs containing functions with arguments but no annotations),
missing methods/attributes, missing classes, and missing submodules. Not to
mention signatures that are impossible to type with the current type system
(you can't actually type zip() or filter(), for example).

We should definitely push back on zealous new converts to typing who insist
that everything should be annotated. But we should also recognize that even
in their current, far from perfect state, type annotations can provide a
lot of value, when used right. (Have you run into VS Code yet? It gets
tremendous value from typing stubs, in the form of improved auto-complete
and hover-doc functionality.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D63B4UHY6I5VUGZPZFYCE62M4DHEKQDH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Jeremiah Vivian]

The code in the pull request is not updated and cannot be updated (though I 
seem to have fixed the bugs in the test in my modified CPython build). It won't 
merge, and I definitely believe it won't pass.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VZSDFTCYQUQUMF37FR4W6XI3C372GHCB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Christopher Barker]

Another concern I have is the over specification of types.

I have seen many examples of, e.g.

func(x: int, y: float, stuff: List(int]):

but very few of:

func(x: SupportsInt, y: SupportsFloat, stuff: Sequence[SupportsInt]):

(or even Iterable[int])

Is that even the right thing to do to get generic number types? How do I
specify a type that could be any number type (float, int, Fraction,
Decimal, numpy.float32) ... I just spent a few minutes perusing the typing
module and MyPy docs, and didn't find a discussion of that.

But I'd really love to see the community as a whole start with more
generic types in examples, and be clear that the concrete types should be
used only when necessary -- even the typing module docs use a lot of
concrete types in the examples.

I think this is an issue for two reasons:

1) Folks learn from examples more than instruction -- people are very
likely to follow the first example they see that seems to do the job.

2) as mentioned in this thread, library authors are being asked to type
hint their libraries. I think it will be all too common for that type
hinting to be more specific than required. For the most part, that won't
cause any real issues right away, but as soon as someone uses that lib in a
large project that enforces static type checking, it's could get ugly.

-CHB






On Tue, Nov 30, 2021 at 1:37 PM Paul Moore  wrote:

> On Tue, 30 Nov 2021 at 19:07, Brett Cannon  wrote:
> >
> > On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano 
> wrote:
> >>
> >> On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:
> >>
> >> > And to be clear, it's often very non-obvious how to annotate something
> >> > - in https://github.com/pfmoore/editables I basically gave up because
> >> > I couldn't work out how to write a maintainable annotation for an
> >> > argument that is "a Path, or something that can be passed to the Path
> >> > constructor to create a Path" (it's essentially impossible without
> >> > copy/pasting the argument annotation for the Path constructor).
> >
> > You're after
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike:
> `str | PathLike[str]` (if you're only accepting string paths).
>
> Well, it's not really. What I'm after, as I stated, is "anything that
> can be passed to the Path constructor". Yes, str | PathLike[str] is
> probably close enough (although why is it OK for me to prohibit bytes
> paths?) but that's what I mean about copying the Path constructor's
> annotations. If Path changes, I have to change my code.
>
> This is a very common idiom:
>
> def f(p: ???):
> p = Path(p)
> ...
>
> Why isn't it correspondingly straightforward to annotate?
>
> If PathLike[str] included str, then it would be a lot easier. It's not
> at all obvious to me why it doesn't (well, that's not entirely true -
> it's because PathLike is an ABC, not a protocol, and it's not intended
> to define "the type of objects that the Path constructor takes"). It
> would still not be documented anywhere, though.
>
> >> I thought that type inference was supposed to solve that sort of
> >> problem? If the typechecker can see that an argument is passed to the
> >> Path constructor, it should be able to infer that it must be the same
> >> types as accepted by Path.
> >
> > I would change that "should" to "may". Python's dynamism makes
> inferencing really hard.
>
> That's fair. That's why I think it should be straightforward for the
> user to explicitly say "this argument should accept the same types as
> pathlib.Path does". If inference can't do it automatically, and the
> user can't (easily) let the checker know that it's OK to do it, then
> we're left with no easy way to express a very common pattern.
>
> >> Aside: I'm a little disappointed in the way the typing ecosystem has
> >> developed. What I understood was that we'd get type inference like ML or
> >> Haskell use, so we wouldn't need to annotate *everything*, only the bits
> >> needed to resolve ambiguity. But what we seem to have got is typing like
> >> C, Pascal and Java, except gradual. Am I being unreasonable to be
> >> disappointed? I'm not a heavy mypy user, I just dabble with it
> >> occasionally, so maybe I've missed something.
> >
> >
> > It really depends on the code base. Type checkers can make guesses based
> on the code they have available to them, but that only works if the usage
> is really clear and the dynamic nature of the code doesn't make things
> murky. For instance, look at open() and how whether you opened a file with
> `b` or not influences whether the object's methods return strings or bytes.
> What would you expect to be inferred in that case if you didn't annotate
> open() with overrides to specify how its arguments influence the returned
> object?
>
> Personally, I'd be quite happy leaving open() as duck typed. I see
> this as what Steven was getting at - the "typing ecosystem" has moved
> into a situation where it's acknowledged that some typing problems are
> really 

[Python-Dev] Re: Expectations of typing (was: The current state of typing PEPs)

[Paul Moore]

On Tue, 30 Nov 2021 at 19:07, Brett Cannon  wrote:
>
> On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano  wrote:
>>
>> On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:
>>
>> > And to be clear, it's often very non-obvious how to annotate something
>> > - in https://github.com/pfmoore/editables I basically gave up because
>> > I couldn't work out how to write a maintainable annotation for an
>> > argument that is "a Path, or something that can be passed to the Path
>> > constructor to create a Path" (it's essentially impossible without
>> > copy/pasting the argument annotation for the Path constructor).
>
> You're after 
> https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike: 
> `str | PathLike[str]` (if you're only accepting string paths).

Well, it's not really. What I'm after, as I stated, is "anything that
can be passed to the Path constructor". Yes, str | PathLike[str] is
probably close enough (although why is it OK for me to prohibit bytes
paths?) but that's what I mean about copying the Path constructor's
annotations. If Path changes, I have to change my code.

This is a very common idiom:

def f(p: ???):
p = Path(p)
...

Why isn't it correspondingly straightforward to annotate?

If PathLike[str] included str, then it would be a lot easier. It's not
at all obvious to me why it doesn't (well, that's not entirely true -
it's because PathLike is an ABC, not a protocol, and it's not intended
to define "the type of objects that the Path constructor takes"). It
would still not be documented anywhere, though.

>> I thought that type inference was supposed to solve that sort of
>> problem? If the typechecker can see that an argument is passed to the
>> Path constructor, it should be able to infer that it must be the same
>> types as accepted by Path.
>
> I would change that "should" to "may". Python's dynamism makes inferencing 
> really hard.

That's fair. That's why I think it should be straightforward for the
user to explicitly say "this argument should accept the same types as
pathlib.Path does". If inference can't do it automatically, and the
user can't (easily) let the checker know that it's OK to do it, then
we're left with no easy way to express a very common pattern.

>> Aside: I'm a little disappointed in the way the typing ecosystem has
>> developed. What I understood was that we'd get type inference like ML or
>> Haskell use, so we wouldn't need to annotate *everything*, only the bits
>> needed to resolve ambiguity. But what we seem to have got is typing like
>> C, Pascal and Java, except gradual. Am I being unreasonable to be
>> disappointed? I'm not a heavy mypy user, I just dabble with it
>> occasionally, so maybe I've missed something.
>
>
> It really depends on the code base. Type checkers can make guesses based on 
> the code they have available to them, but that only works if the usage is 
> really clear and the dynamic nature of the code doesn't make things murky. 
> For instance, look at open() and how whether you opened a file with `b` or 
> not influences whether the object's methods return strings or bytes. What 
> would you expect to be inferred in that case if you didn't annotate open() 
> with overrides to specify how its arguments influence the returned object?

Personally, I'd be quite happy leaving open() as duck typed. I see
this as what Steven was getting at - the "typing ecosystem" has moved
into a situation where it's acknowledged that some typing problems are
really hard, due to Python's dynamism, and yet there's still a drive
to try to express such highly dynamic type constraints statically. And
worse still, to insist that doing so is somehow necessary.

Whatever happened to "practicality beats purity", and typing being
"gradual"? Surely annotating everything except open() is practical and
gradual?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YXVXN22TIHBT454CKHKKOCELANF73FZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Expectations of typing (was: The current state of typing PEPs)

[Brett Cannon]

On Tue, Nov 30, 2021 at 9:09 AM Steven D'Aprano  wrote:

> On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:
>
> > And to be clear, it's often very non-obvious how to annotate something
> > - in https://github.com/pfmoore/editables I basically gave up because
> > I couldn't work out how to write a maintainable annotation for an
> > argument that is "a Path, or something that can be passed to the Path
> > constructor to create a Path" (it's essentially impossible without
> > copy/pasting the argument annotation for the Path constructor).
>

You're after
https://docs.python.org/3/library/os.html?highlight=pathlike#os.PathLike:
`str | PathLike[str]` (if you're only accepting string paths).


>
> I thought that type inference was supposed to solve that sort of
> problem? If the typechecker can see that an argument is passed to the
> Path constructor, it should be able to infer that it must be the same
> types as accepted by Path.
>

I would change that "should" to "may". Python's dynamism makes inferencing
really hard.


>
> Aside: I'm a little disappointed in the way the typing ecosystem has
> developed. What I understood was that we'd get type inference like ML or
> Haskell use, so we wouldn't need to annotate *everything*, only the bits
> needed to resolve ambiguity. But what we seem to have got is typing like
> C, Pascal and Java, except gradual. Am I being unreasonable to be
> disappointed? I'm not a heavy mypy user, I just dabble with it
> occasionally, so maybe I've missed something.
>

It really depends on the code base. Type checkers can make guesses based on
the code they have available to them, but that only works if the usage is
really clear and the dynamic nature of the code doesn't make things murky.
For instance, look at open() and how whether you opened a file with `b` or
not influences whether the object's methods return strings or bytes. What
would you expect to be inferred in that case if you didn't annotate open()
with overrides to specify how its arguments influence the returned object?

It also depends on how lenient your type checker is willing to be. Compiled
languages get to know all potential inputs and outputs of a function
upfront, while Python it's a guess based on what you happen to have
installed since you can always just `importlib.import_module()` anything at
any time. But since type checkers typically prefer false-positives over
false-negatives for correctness, it means having to clarify more code.


>
>
>
> > Anyway, we're *way* off topic now, and I doubt there's much that the
> > SC or python-dev can do to change the community view on typing,
> > anyway.
>
> Heh. We could update PEP 8 to ban type annotations, then watch as the
> people who over-zealously apply PEP 8 to everything AND over-zealously
> insist on adding type annotations to everything have their heads
> explode.
>
>
> --
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Y6GOWIZV5JCOG5TP4ZZ4SVLXL4BGDJTI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PTIX4SV7OECBKAGXPAOHSGVPMZNN47B4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum  wrote:
> How about *not* asking for an exception and just following the PEP 387 
> process? Is that really too burdensome?

The Backward Compatibility section gives an explanation:

"This change does not follow the PEP 387 deprecation process. There is no
known way to emit a deprecation warning when a macro is used as a
l-value, but not when it's used differently (ex: r-value)."

Apart of compiler warnings, one way to implement the PEP 387
"deprecation process" would be to announce the change in two "What's
New in Python 3.X?" documents. But I expect that it will not be
efficient. Extract of the Rejected Idea section:

"(...) only few developers read the documentation, and only a minority
is tracking changes of the Python C API documentation."

In my experience, even if a DeprecationWarning is emitted at runtime,
developers miss or ignore it. See the recent "[Python-Dev] Do we need
to remove everything that's deprecated?" discussion and complains
about recent removal of deprecated features, like:

* collections.MutableMapping was deprecated for 7 Python versions
(deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11
* the "U" open() flag was deprecated for 10 Python versions
(deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11

For this specific PEP changes, I consider that the number of impacted
projects is low enough to skip a deprecation process: only 4 projects
are known to be impacted. One year ago (Python 3.10), 16 were
impacted, and 12 have already been updated in the meanwhile. I'm
talking especially about Py_TYPE() and Py_SIZE() changes which, again,
has been approved by the Steering Council.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AAQCCRSAKBAPCKWWT5BQTBPN7TL7WHVJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Ethan Furman]

On 11/26/21 1:13 AM, Paul Moore wrote:
> On Fri, 26 Nov 2021 at 05:14, Guido van Rossum wrote:
>>
>> My memory is also hazy, but I'm quite sure that *in my mind* annotations were
>> intended as a compromise between conflicting proposals for *typing*. We 
didn't
>> have agreement on the syntax or semantics, but we did know we wanted to do
>> something with types eventually.
>
> More hazy memories here, but I think the original proposal left open
> the possibility of annotations not being types at all - for example,
> being docstrings for the arguments, or option names for a "function
> call to CLI" tool, etc. I think that some libraries took this approach
> (in particular the "CLI builder" idea).

I have one such tool.  Fortunately for me, I strongly dislike having the annotations inside the function header as it 
quickly gets difficult for me to read, so I was already using decorator syntax for the annotations.  When PEP 484 was 
accepted I just changed where the annotations were being stored from __annotations__ to my own dunder.


--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VC6NSJ4MXJOTNBKO7OOBSV7JHQ5JBPLO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Guido van Rossum]

How about *not* asking for an exception and just following the PEP 387
process? Is that really too burdensome?

On Tue, Nov 30, 2021 at 10:30 AM Victor Stinner  wrote:

> Hi,
>
> I propose to disallow using macros as l-value.  Read and comment the
> plain text below, or read the PEP 674 online,
> https://python.github.io/peps/pep-0674/
>
> While I'm not sure that the proposed changes are really controversial,
> I decided to write a formal PEP since the incompatible changes are not
> following the PEP 387 deprecation process and so a format PEP 387
> exception is better. Well, the list of modified macros is also quite
> long. Moreover, a PEP is a way to document and announce the changes
> ;-)
>
> The Py_TYPE() and Py_SIZE() changes are already approved the Steering
> Council, but I prefer to list them in the PEP:
> https://github.com/python/steering-council/issues/79
>
> In practice, I'm only aware of 4 projects impacted by these changes,
> and I wrote the pythoncapi_compat project which updates automatically
> C extensions: add Python 3.11 support, without losing support for
> older Python versions. I already prepared major projects like Cython
> and numpy for these changes (in total, 14 impacted projects have
> already been updated).
>
> Victor
>
> ---
>
> PEP: 674
> Title: Disallow using macros as l-value
> Author: Victor Stinner 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 30-Oct-2021
> Python-Version: 3.11
>
> Abstract
> 
>
> Incompatible C API change disallowing using macros as l-value to allow
> evolving CPython internals and to ease the C API implementation on other
> Python implementation.
>
> In practice, the majority of projects impacted by these incompatible
> changes should only have to make two changes:
>
> * Replace ``Py_TYPE(obj) = new_type;``
>   with ``Py_SET_TYPE(obj, new_type);``.
> * Replace ``Py_SIZE(obj) = new_size;``
>   with ``Py_SET_SIZE(obj, new_size);``.
>
>
> Rationale
> =
>
> Using a macro as a l-value
> --
>
> In the Python C API, some functions are implemented as macro because
> writing a macro is simpler than writing a regular function. If a macro
> exposes directly a struture member, it is technically possible to use
> this macro to not only get the structure member but also set it.
>
> Example with the Python 3.10 ``Py_TYPE()`` macro::
>
> #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)
>
> This macro can be used as a **r-value** to **get** an object type::
>
> type = Py_TYPE(object);
>
> It can also be used as **l-value** to **set** an object type::
>
> Py_TYPE(object) = new_type;
>
> It is also possible to set an object reference count and an object size
> using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.
>
> Setting directly an object attribute relies on the current exact CPython
> implementation. Implementing this feature in other Python
> implementations can make their C API implementation less efficient.
>
> CPython nogil fork
> --
>
> Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch
> `_. This fork has no
> ``PyObject.ob_refcnt`` member, but a more elaborated implementation for
> reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
> fails with a compiler error.
>
> Merging the nogil fork into the upstream CPython main branch requires
> first to fix this C API compatibility issue. It is a concrete example of
> a Python optimization blocked indirectly by the C API.
>
> This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
> has been already modified to disallow using it as a l-value.
>
> HPy project
> ---
>
> The `HPy project `_ is a brand new C API for
> Python using only handles and function calls: handles are opaque,
> structure members cannot be accessed directly,and pointers cannot be
> dereferenced.
>
> Disallowing the usage of macros as l-value helps the migration of
> existing C extensions to HPy by reducing differences between the C API
> and the HPy API.
>
> PyPy cpyext module
> --
>
> In PyPy, when a Python object is accessed by the Python C API, the PyPy
> ``cpyext`` module has to convert PyPy object to a CPython object. While
> PyPy objects are designed to be efficient with the PyPy JIT compiler,
> CPython objects are less efficient and increase the memory usage.
>
> This PEP alone is not enough to get rid of the CPython objects in the
> PyPy ``cpyext`` module, but it is a step towards this long term goal.
> PyPy already supports HPy which is a better solution in the long term.
>
>
> Specification
> =
>
> Disallow using macros as l-value
> 
>
> PyObject and PyVarObject macros
> ^^^
>
> * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
> * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead
>
> "GET" macros
> 
>
> * 

[Python-Dev] PEP 674: Disallow using macros as l-value

[Victor Stinner]

Hi,

I propose to disallow using macros as l-value.  Read and comment the
plain text below, or read the PEP 674 online,
https://python.github.io/peps/pep-0674/

While I'm not sure that the proposed changes are really controversial,
I decided to write a formal PEP since the incompatible changes are not
following the PEP 387 deprecation process and so a format PEP 387
exception is better. Well, the list of modified macros is also quite
long. Moreover, a PEP is a way to document and announce the changes
;-)

The Py_TYPE() and Py_SIZE() changes are already approved the Steering
Council, but I prefer to list them in the PEP:
https://github.com/python/steering-council/issues/79

In practice, I'm only aware of 4 projects impacted by these changes,
and I wrote the pythoncapi_compat project which updates automatically
C extensions: add Python 3.11 support, without losing support for
older Python versions. I already prepared major projects like Cython
and numpy for these changes (in total, 14 impacted projects have
already been updated).

Victor

---

PEP: 674
Title: Disallow using macros as l-value
Author: Victor Stinner 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Oct-2021
Python-Version: 3.11

Abstract


Incompatible C API change disallowing using macros as l-value to allow
evolving CPython internals and to ease the C API implementation on other
Python implementation.

In practice, the majority of projects impacted by these incompatible
changes should only have to make two changes:

* Replace ``Py_TYPE(obj) = new_type;``
  with ``Py_SET_TYPE(obj, new_type);``.
* Replace ``Py_SIZE(obj) = new_size;``
  with ``Py_SET_SIZE(obj, new_size);``.


Rationale
=

Using a macro as a l-value
--

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
exposes directly a struture member, it is technically possible to use
this macro to not only get the structure member but also set it.

Example with the Python 3.10 ``Py_TYPE()`` macro::

#define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)

This macro can be used as a **r-value** to **get** an object type::

type = Py_TYPE(object);

It can also be used as **l-value** to **set** an object type::

Py_TYPE(object) = new_type;

It is also possible to set an object reference count and an object size
using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.

Setting directly an object attribute relies on the current exact CPython
implementation. Implementing this feature in other Python
implementations can make their C API implementation less efficient.

CPython nogil fork
--

Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch
`_. This fork has no
``PyObject.ob_refcnt`` member, but a more elaborated implementation for
reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
fails with a compiler error.

Merging the nogil fork into the upstream CPython main branch requires
first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
has been already modified to disallow using it as a l-value.

HPy project
---

The `HPy project `_ is a brand new C API for
Python using only handles and function calls: handles are opaque,
structure members cannot be accessed directly,and pointers cannot be
dereferenced.

Disallowing the usage of macros as l-value helps the migration of
existing C extensions to HPy by reducing differences between the C API
and the HPy API.

PyPy cpyext module
--

In PyPy, when a Python object is accessed by the Python C API, the PyPy
``cpyext`` module has to convert PyPy object to a CPython object. While
PyPy objects are designed to be efficient with the PyPy JIT compiler,
CPython objects are less efficient and increase the memory usage.

This PEP alone is not enough to get rid of the CPython objects in the
PyPy ``cpyext`` module, but it is a step towards this long term goal.
PyPy already supports HPy which is a better solution in the long term.


Specification
=

Disallow using macros as l-value


PyObject and PyVarObject macros
^^^

* ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
* ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead

"GET" macros


* ``PyByteArray_GET_SIZE()``
* ``PyBytes_GET_SIZE()``
* ``PyCFunction_GET_CLASS()``
* ``PyCFunction_GET_FLAGS()``
* ``PyCFunction_GET_FUNCTION()``
* ``PyCFunction_GET_SELF()``
* ``PyCell_GET()``
* ``PyCode_GetNumFree()``
* ``PyDict_GET_SIZE()``
* ``PyFunction_GET_ANNOTATIONS()``
* ``PyFunction_GET_CLOSURE()``
* ``PyFunction_GET_CODE()``
* ``PyFunction_GET_DEFAULTS()``
* ``PyFunction_GET_GLOBALS()``
* 

[Python-Dev] Re: The current state of typing PEPs

[Christopher Barker]

This is great Patrick, thanks.

My use case is similar to Patrick's, except it builds on dataclasses
directly:

It's part of a larger system, but I've just pulled it out into its own
poorly documented and poorly tested package:

https://github.com/PythonCHB/flexi

(I think it's generally useful, so may one day make a proper package out of
it)

Anyway, the principle is this:

The goal is to have a flexible data model that enforces the structure, but
not necessarily the validity of values, or even types. This is useful
because we need to store / edit / import complex hierarchy of data, and we
very much don't want to have all of it to have to be valid in order to even
store it. An example is a dataset we recently imported (from
spreadsheets--arrgghh), where in a thousand records, they had put "too
viscous to measure" instead of a value for interfacial tension. That, of
course, would be helpful to a person reading the spreadsheet, but would
have been a pain if we had enforced that that field had to be a float > 0.
Instead, it imported fine, and then our validation code flagged the issue
to be fixed later. And all the rest of the data could be used in the
meantime.

Anyway, that was a digression (but explains why we don't use Pydantic).
Functionally, we need the "nodes" in the data model to be able to convert
themselves to/from JSON compatible Python, and validate themselves, etc.
This is done by using the actual type object, as stored in
dataclasses.Field.type

This has worked nicely for us, and dataclasses saved us a lot of
boilerplate code.

if we do

from future import Annotations

then the system breaks, as we have a string instead of the actual type that
is needed.

Solutions:

I *think* PEP 649 would work fine for us, as the annotation would get
evaluated when it was added to the field object.

But inspect.get_annotations isn't really helpful for this use, as the
Field.type attributes aren't annotations anymore. So we would have to
eval() the strings ourselves, and I think it would be unclear what
namespaces to use for that.

If dataclasses were to use

inspect.get_annotations(cls, eval_str=rue)

in order to extract the types to put into the Field objects, then I think
all would be good for us. And as I understand it, dataclasses itself
doesn't do anything with that attribute anyway.

If that's not decided to be a good thing for dataclasses, then I think we'd
have to re-implement  a lot ourselves, though it could probably be hacked
in - I haven't tried that yet.

BTW, I don't think we've heard from the attrs (and especially cattrs) folks
in this thread. A search of the repo issues indicates that there has been
some discussion of PEP 563's impact, but it's not totally clear to me if
it's been resolved.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZYWLGMP5UO7NTABVJJOHN7ZSS4KUM6UG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Steven D'Aprano]

On Tue, Nov 30, 2021 at 02:30:18PM +, Paul Moore wrote:

> And to be clear, it's often very non-obvious how to annotate something
> - in https://github.com/pfmoore/editables I basically gave up because
> I couldn't work out how to write a maintainable annotation for an
> argument that is "a Path, or something that can be passed to the Path
> constructor to create a Path" (it's essentially impossible without
> copy/pasting the argument annotation for the Path constructor).

I thought that type inference was supposed to solve that sort of 
problem? If the typechecker can see that an argument is passed to the 
Path constructor, it should be able to infer that it must be the same 
types as accepted by Path.

Aside: I'm a little disappointed in the way the typing ecosystem has 
developed. What I understood was that we'd get type inference like ML or 
Haskell use, so we wouldn't need to annotate *everything*, only the bits 
needed to resolve ambiguity. But what we seem to have got is typing like 
C, Pascal and Java, except gradual. Am I being unreasonable to be 
disappointed? I'm not a heavy mypy user, I just dabble with it 
occasionally, so maybe I've missed something.



> Anyway, we're *way* off topic now, and I doubt there's much that the
> SC or python-dev can do to change the community view on typing,
> anyway.

Heh. We could update PEP 8 to ban type annotations, then watch as the 
people who over-zealously apply PEP 8 to everything AND over-zealously 
insist on adding type annotations to everything have their heads 
explode.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Y6GOWIZV5JCOG5TP4ZZ4SVLXL4BGDJTI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Patrick Arminio]

On Fri, 26 Nov 2021 at 09:14, Paul Moore  wrote:

> I'd therefore interpret Barry's plea as being for *anyone* with a use
> for annotations to provide their feedback (at least, anyone who
> accepts that annotations are types), with particular emphasis on
> people who want to use the types declared in annotations to affect
> runtime behaviour, as that's the most under-represented group at the
> moment (and it's not clear whether it's under-represented because
> there aren't many such uses, or because the users aren't being heard
> from).


I don't know if this helps, but I'm a maintainer/creator of a Python library
that uses type annotations at runtime. The library is used to create GraphQL
APIs[1]. We took a lot of inspiration from dataclasses, our API looks like
this:

```python
@strawberry.type
class User:
name: str
```

The `strawberry.type` decorator reads the annotations and creates an object
with a list
of fields and types, but it doesn't try to resolve them (that happens later
to prevent issues
with circular dependencies and so on).

I remember the discussion around PEP 563 and how it would break libraries
using type annotations,
in our case things are fine since the types we define should be defined in
the global namespace[2]
anyway. I think the only issue we have with type annotations is when you
have circular references
inside your python code (where you'd use `if TYPE_CHECKING: import X`), we
"solved" that with
a custom class called `LazyType[type, module]`, that allows to specify a
type and where it comes
from, so our library can resolve the proper type, even if it was not
imported in the namespace of the
class using that type.

Sorry for the digression into some of the details of how the library works,
I hope it helps a bit
understanding the use case.

I think there's definitely use cases for type hints being used at runtime.
I personally love
the fact that we are allowed to do this, in other languages (like
TypeScript) you don't have
access to the types at runtime and it does prevent the creation of
libraries like dataclasses
or mine, which is a bit of a shame (you'd need to use code generation to
get a similar experience).

There's also other libraries that use type hints at runtime (not for
runtime type validation[3]):

- Typer - http://typer.tiangolo.com
- SQLModel - http://sqlmodel.tiangolo.com
- Beanie - https://github.com/roman-right/beanie/
- Pydantic - http://pydantic-docs.helpmanual.io
- odmantic - https://github.com/art049/odmantic/
- singledispatch -
https://docs.python.org/3/library/functools.html#functools.singledispatch


[1] If you're not familiar with GraphQL there's some info here:
https://graphql.org
[2] I did ask if there was a way to resolve types when having them
encapsulated into functions,
but that was a specific use case with tests and I was/am fine with
using global there

https://mail.python.org/archives/list/typing-...@python.org/thread/SNKJB2U5S74TWGDWVD6FMXOP63WVIGDR/#A5JFR7RDE7NNKESM2DWYM3ZT5ZYGOLCF

[3] while I see runtime validation being something useful, I think it is
worth taking a look at
   different use cases for type hints :)

-- 
Patrick Arminio
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ODJYIGGSHMNS3BZND7UNVROPQ4LPW4R7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 12:39, Oscar Benjamin  wrote:

> There are other open "issues" like this for SymPy where the
> presumption is that not having type hints is now to be considered a
> deficiency of the library regardless of whether the hints have any
> benefit for internal use. I don't object to adding the hints but it's
> a huge amount of work that someone would have to do and I don't want
> to add useless/inaccurate hints temporarily (as some have suggested to
> do).

This is precisely the sort of concern I have. Although it is awfully
tempting to passive-aggressively annotate everything as Any, just to
shut people up :-(

And to be clear, it's often very non-obvious how to annotate something
- in https://github.com/pfmoore/editables I basically gave up because
I couldn't work out how to write a maintainable annotation for an
argument that is "a Path, or something that can be passed to the Path
constructor to create a Path" (it's essentially impossible without
copy/pasting the argument annotation for the Path constructor). I also
spent a lot of time trying to deal with a typeshed bug. And even
worse, if I put the types in a .pyi file (which I want to do, because
I don't want the library to import typing at runtime because it's a
significant overhead for something that's supposed to be lightweight),
apparently mypy won't check them so I gain no benefit for the project
itself.

Anyway, we're *way* off topic now, and I doubt there's much that the
SC or python-dev can do to change the community view on typing,
anyway.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4DUJM7MNJSJ3TOYYFILQCG4RT4CU6DME/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Walter Dörwald]

On 29 Nov 2021, at 23:56, Barry Warsaw wrote:


[...]
(not that you're not allowed to use for anything else, of course you 
are, but that other uses won;t be taken into account when designing 
the new interface)


But I have never seen that clearly stated anywhere. The closest is 
from PEP 563, where it says:


"""
With this in mind, uses for annotations incompatible with the 
aforementioned PEPs should be considered deprecated.

"""

Which pretty much makes the point, but it's a bit subtle -- what does 
"incompatible' mean?


You make a good point.  I agree that while all the signs are there for 
“annotations are for typing”, this has never been explicitly or 
sufficiently codified, and I’ve been proposing that Someone write a 
PEP that makes this an official pronouncement.  That someone may very 
well be a person on the SC, but given the timing of elections, it’s 
likely to fall to the next SC, if they still agree with that position! 
:-D


My recollection of the history of annotations falls somewhere between 
Greg’s and Guido’s.  Annotations as a feature were inspired by the 
typing use case (with no decision at the time whether those were to be 
static or runtime checks), but at the same time allowing for 
experimentation for other use cases.  Over time, 
annotations-for-typing clearly won the mindset and became the 
predominant use case.  Personally, I was strongly against type 
annotations because of my experience in other languages, but over time 
I was also won over.  From library documentation, to complex code 
bases, to the transient nature of contributors, type annotations are 
pretty compelling, and most (but not all) of my worries really 
didn’t come to fruition.


We can lament the non-typing use of annotations, but I think that 
horse is out of the barn and I don’t know how you would resolve 
conflicts of use for typing and non-typing annotations.  It’s been a 
slow boil, with no definitive pronouncement, and that needs to be 
fixed, but I think it's just acknowledging reality.  That’s my 
personal opinion.


But isn't that the reason why we have `typing.Annotated`, so that 
annotations used for typing and annotations used for other purposes can 
coexist?


An annotation used for typing only is:

```

def f(x: int):

...return x
...

f.__annotations__['x']


```

An annotation used for something else is:

```

def f(x: 'something not typing related'):

...return x
...

f.__annotations__['x']

'something not typing related'
```

`typing.Annotated` gives us both:

```

from typing import *
def f(x: Annotated[int, 'something not typing related']):

...return x
...

f.__annotations__['x'].__args__[0]



f.__annotations__['x'].__metadata__[0]

'something not typing related'
```

Granted, for someone who only wants to use annotations for their own 
purpose, digging out the original not typing related stuff from 
`typing.Annotated` is more work, but doable.


Or am I missing something here?


[...]


Servus,
   Walter
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4WM43CS45R7H7X2WO4F3UVDCH7HFHDJJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Steven D'Aprano]

On Tue, Nov 30, 2021 at 09:17:13AM +, Paul Moore wrote:

> Also, related to the question Terry raised, IMO it would be useful to
> have a clear statement on code that *does* use type annotations, but
> violates them at runtime. To be specific, is the following considered
> as an error?
> 
> >>> def muladd(x: int, y: int, z: int) -> int:
> ... return x * (y+z)
> ...
> >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447

Isn't that just duck-typing? You've got a function that is documented as 
taking arguments of one type (int), but actually will work with any 
numeric type (and some non-numeric types) that quacks like an int.

muladd(2, 'ab', 'cd')  # returns 'abcdabcd'

If you replaced the type annotations with a docstring that listed the 
parameters x, y, z as int, would duck-typing be wrong? Maybe.

I don't think we can make any concrete claims here. Sometimes 
duck-typing is perfectly fine. Sometimes its not. It depends on the 
function's implementation and its semantics. Sometimes calling a 
function with a duck-typed value seems to work fine but the result is 
meaningless junk. (Garbage In, Garbage Out.)

I guess all we can say is that the Python language is agnostic and 
neutral on this matter.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W24VGDUH6F43C6VT4OUGERNEZFW223MP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Oscar Benjamin]

On Tue, 30 Nov 2021 at 09:23, Paul Moore  wrote:
>
> On Tue, 30 Nov 2021 at 02:52, Steve Dower  wrote:
> >
> > THAT'S the kind of thing that also has been happening with typing, and
> > why some of us feel the need to publicly re-state things that are all
> > agreed upon within this group, but are struggling to be heard over the
> > public discourse on the topics. And this kind of reiteration is easier
> > when our official documents (PEPs, etc.) state it explicitly.
>
> This is very definitely the case. There's a subtle (maybe not so
> subtle, actually) and increasing pressure on projects to add typing.
> Often with little or no justification beyond "you should", as if
> having typing is a sort of "obvious best practice". Sometimes "because
> it will make it easier for your users who use typing" is given as a
> justification, but while that's fair, it's also a disturbing gradual
> pressure for typing to extend everywhere, manifesting by making it
> feel like not adding typing is somehow "not caring about your users".

I suspect that a big source of this is editors that use typing for
autocompletion etc. Speaking as a vim user this is something that has
passed me by except that I have students who I see adding type hints
even though I deliberately haven't taught them anything about type
hints. I can only presume that some editor is doing this for them or
telling them that they need to do this (students often can't tell the
difference between editor warnings and actual errors).

Others have mentioned the pressure on libraries to adopt typing and
I've certainly noticed this with SymPy. I think type hints could be
good for SymPy for internal use but it seems that a lot of users want
it for external reasons that I don't always understand but that also
seems to come partly from editors as well.

Some people apparently want to add type hints that look completely
useless to me like
def f() -> Union[MyClass, Any]
As I understand it this does not give any meaningful information to a
type checker but it apparently makes vscode work better:
https://github.com/sympy/sympy/pull/22180#discussion_r718917577

Here in SymPy you can see a suggestion that type hints are needed
because otherwise vscode is slow and uses 4GB of memory just to load
documentation for SymPy functions:
https://github.com/sympy/sympy/issues/17945#issuecomment-928798977

Not to do with editors but also indicative is this PR whose title
implies that SymPy is "incompatible" with PEP 561 (a PEP that I had
never heard of before seeing the PR):
https://github.com/sympy/sympy/pull/22337

There are other open "issues" like this for SymPy where the
presumption is that not having type hints is now to be considered a
deficiency of the library regardless of whether the hints have any
benefit for internal use. I don't object to adding the hints but it's
a huge amount of work that someone would have to do and I don't want
to add useless/inaccurate hints temporarily (as some have suggested to
do).

--
Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQ773X24AICPLPLIN742B25WNVZJHMRO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 09:37, Chris Angelico  wrote:
>
> On Tue, Nov 30, 2021 at 8:19 PM Paul Moore  wrote:
> > Also, related to the question Terry raised, IMO it would be useful to
> > have a clear statement on code that *does* use type annotations, but
> > violates them at runtime. To be specific, is the following considered
> > as an error?
> >
> > >>> def muladd(x: int, y: int, z: int) -> int:
> > ... return x * (y+z)
> > ...
> > >>> muladd(3.1459, 87.33, 2.7e2)
> > 1124.124447
> >
>
> My understanding is that it's precisely as wrong, or not wrong, as this:
>
> def add(x, y):
>"""Multiply two numbers"""
> return x / y
>
> To my mind, annotations are machine-readable metadata, with no
> inherent "correctness" to them (from the language's point of view),
> other than syntactically.

That makes sense to me, and I'd support that being clearly stated as
the "official position".

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LUY2SXBGX4TQKTX6PE2FHYHABDCYY2IN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Chris Angelico]

On Tue, Nov 30, 2021 at 8:19 PM Paul Moore  wrote:
> Also, related to the question Terry raised, IMO it would be useful to
> have a clear statement on code that *does* use type annotations, but
> violates them at runtime. To be specific, is the following considered
> as an error?
>
> >>> def muladd(x: int, y: int, z: int) -> int:
> ... return x * (y+z)
> ...
> >>> muladd(3.1459, 87.33, 2.7e2)
> 1124.124447
>

My understanding is that it's precisely as wrong, or not wrong, as this:

def add(x, y):
   """Multiply two numbers"""
return x / y

To my mind, annotations are machine-readable metadata, with no
inherent "correctness" to them (from the language's point of view),
other than syntactically.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54VRD5SJ3RQT6W4AX5WPIW2OUWZNWPZO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The current state of typing PEPs

[Paul Moore]

On Tue, 30 Nov 2021 at 02:52, Steve Dower  wrote:
>
> THAT'S the kind of thing that also has been happening with typing, and
> why some of us feel the need to publicly re-state things that are all
> agreed upon within this group, but are struggling to be heard over the
> public discourse on the topics. And this kind of reiteration is easier
> when our official documents (PEPs, etc.) state it explicitly.

This is very definitely the case. There's a subtle (maybe not so
subtle, actually) and increasing pressure on projects to add typing.
Often with little or no justification beyond "you should", as if
having typing is a sort of "obvious best practice". Sometimes "because
it will make it easier for your users who use typing" is given as a
justification, but while that's fair, it's also a disturbing gradual
pressure for typing to extend everywhere, manifesting by making it
feel like not adding typing is somehow "not caring about your users".

Also, related to the question Terry raised, IMO it would be useful to
have a clear statement on code that *does* use type annotations, but
violates them at runtime. To be specific, is the following considered
as an error?

>>> def muladd(x: int, y: int, z: int) -> int:
... return x * (y+z)
...
>>> muladd(3.1459, 87.33, 2.7e2)
1124.124447

If (as in the current interpreter) typing is optional, this isn't an
error. Certainly, running mypy over it fails, but the point here is
that mypy is an external tool - the Python interpreter itself (and the
language definition) allows it. I think it would be useful to have a
clear message that this is intended, and won't be altered lightly.
Remember that the "correct" annotation here is either muladd(x: Any,
y: Any, z: Any) -> Any, or muladd(x: SomeMultiplyProtocol, y:
SomeAddProtocol, z: SomeAddProtocol) -> Any, neither of which is
particularly helpful...

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E36ILTCKFVW3MNFU753PKKWIFY2FWQXU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Optimizing literal comparisons and contains

[Victor Stinner]

If someone wants to experiment such optimization, there is no need to
modify the Python internal optimizer, it can be done externally:
https://faster-cpython.readthedocs.io/ast_optimizer.html

For example, I implemented many optimizations like constant
propagation and loop unrolling in my old AST fatoptimizer project:
https://github.com/vstinner/fatoptimizer/blob/master/doc/optimizations.rst

Optimizing "2 < 3" becomes more interesting when other optimizations
are implemented, like constant propagation or loop unrolling. So "x=2;
if x < 10: ..." can be optimized for example.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DGJLWZ6TCLFC3KLN7KZ7ALMLB5QD6FSY/
Code of Conduct: http://python.org/psf/codeofconduct/