[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Larry Hastings]

On 4/15/21 9:24 PM, Inada Naoki wrote:

Unlike simple function case, PEP 649 creates function object instead
of code object for __co_annotation__ of methods.
It cause this overhead.  Can we avoid creating functions for each annotation?



As the implementation of PEP 649 currently stands, there are two reasons 
why the compiler might pre-bind the __co_annotations__ code object to a 
function, instead of simply storing the code object:


 * If the annotations refer to a closure ("freevars" is nonzero), or
 * If the annotations /possibly/ refer to a class variable (the
   annotations code object contains either LOAD_NAME or LOAD_CLASSDEREF).

If the annotations refer to a closure, then the code object also needs 
to be bound with the "closure" tuple.  If the annotations possibly refer 
to a class variable, then the code object also needs to be bound with 
the current "f_locals" dict.  (Both could be true.)


Unfortunately, when generating annotations on a method, references to 
builtins (e.g. "int", "str") seem to generate LOAD_NAME instructions 
instead of LOAD_GLOBAL.  Which means pre-binding the function happens 
pretty often for methods.  I believe in your benchmark it will happen 
every time.  There's a lot of code, and a lot of runtime data 
structures, inside compile.c and symtable.c behind the compiler's 
decision about whether something is NAME vs GLOBAL vs DEREF etc, and I 
wasn't comfortable with seeing if I could fix it.


Anyway I assume it wasn't "fixable".  The compiler would presumably 
already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL 
would be cheaper every time for a global or builtin.  The fact that it 
already doesn't do so implies that it can't.



At the moment I have only one idea for a possible optimization, as 
follows.  Instead of binding the function object immediately, it /might/ 
be cheaper to write the needed values into a tuple, then only actually 
bind the function object on demand (like normal).


I haven't tried this because I assumed the difference at runtime would 
be negligible.  On one hand, you're creating a function object; on the 
other you're creating a tuple.  Either way you're creating an object at 
runtime, and I assumed that bound functions weren't /that/ much more 
expensive than tuples.  Of course I could be very wrong about that.


The other thing is, it would be a lot of work to even try the 
experiment.  Also, it's an optimization, and I was more concerned with 
correctness... and getting it done and getting this discussion underway.



What follows are my specific thoughts about how to implement this 
optimization.


In this scenario, the logic in the compiler that generates the code 
object would change to something like this:


   has_closure = co.co_freevars != 0
   has_load_name = co.co_code does not contain LOAD_NAME or
   LOAD_CLASSDEREF bytecodes
   if not (has_closure or has_load_name):
    co_ann = co
   elif has_closure and (not has_load_name):
    co_ann = (co, freevars)
   elif (not has_closure) and has_load_name:
    co_ann = (co, f_locals)
   else:
    co_ann = (co, freevars, f_locals)
   setattr(o, "__co_annotations__", co_ann)

(The compiler would have to generate instructions creating the tuple and 
setting its members, then storing the resulting object on the object 
with the annotations.)


Sadly, we can't pre-create this "co_ann" tuple as a constant and store 
it in the .pyc file, because the whole point of the tuple is to contain 
one or more objects only created at runtime.



The code implementing __co_annotations__ in the three objects (function, 
class, module) would examine the object it got.  If it was a code 
object, it would bind it; if it was a tuple, it would unpack the tuple 
and use the values based on their type:


   // co_ann = internal storage for __co_annotations__
   if isinstance(co_ann, FunctionType) or (co_ann == None):
    return co_ann
   co, freevars, locals = None
   if isinstance(co_ann, CodeType):
    co = co_ann
   else:
    assert isinstance(co_ann, tuple)
    assert 1 <= len(co_ann) <= 3
    for o in co_ann:
        if isinstance(o, CodeObject):
            assert not co
    co = o
        elif isinstance(o, tuple):
            assert not freevars
    freevars = o
        elif isinstance(o, dict):
            assert not locals
    locals = o
        else:
            raise ValueError(f"illegal value in co_annotations
   tuple: {o!r}")
   co_ann = make_function(co, freevars=freevars, locals=locals)
   return co_ann


If you experiment with this approach, I'd be glad to answer questions 
about it, either here or on Github, etc.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived 

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Inada Naoki]

>
> I will read PEP 649 implementation to find missing optimizations other
> than GH-25419 and GH-23056.
>

I found each "__co_annotation__" has own name like "func0.__co_annotation__".
It increased pyc size a little.
I created a draft pull request for cherry-picking GH-25419 and
GH-23056 and using just "__co_annotation__" as a name.
https://github.com/larryhastings/co_annotations/pull/9/commits/48a99e0aafa2dd006d72194bc1d7d47443900502

```
# previous result
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Use single name
$ ./python ~/ann_test.py 3
code size: 182088 bytes
memory: 209257 bytes
unmarshal: avg: 539.841ms +/-0.227ms
exec: avg: 98.351ms +/-0.064ms
```

It reduced code size and unmarshal time by 10%.
I confirmed GH-25419 and GH-23056 works very well. All same constants
are shared.

Unmarshal time is still slow. It is caused by unmarshaling code object.
But I don't know where is the bottleneck: parsing marshal file, or
creating code object.

---

Then, I tried to measure method annotation overhead.
Code: 
https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d#file-ann_test_method-py
Result:

```
# No annotation
$ ./python ~/ann_test_method.py 0
code size: 113019 bytes
memory: 256008 bytes
unmarshal: avg: 336.665ms +/-6.185ms
exec: avg: 176.791ms +/-3.067ms

# PEP 563
$ ./python ~/ann_test_method.py 2
code size: 120532 bytes
memory: 269004 bytes
unmarshal: avg: 348.285ms +/-0.102ms
exec: avg: 176.933ms +/-4.343ms

# PEP 649 (all optimization included)
$ ./python ~/ann_test_method.py 3
code size: 196135 bytes
memory: 436565 bytes
unmarshal: avg: 579.680ms +/-0.147ms
exec: avg: 259.781ms +/-7.087ms
```

PEP 563 vs 649
* code size: +63%
* memory: +62%
* import time: +60%

PEP 649 annotation overhead (compared with no annotation):
* code size: +83 byte/method
* memory: +180 byte/method
* import time: +326 us/method

It is disappointing because having thousands methods is very common
for web applications.

Unlike simple function case, PEP 649 creates function object instead
of code object for __co_annotation__ of methods.
It cause this overhead.  Can we avoid creating functions for each annotation?

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4KUXC373ZOP7YHZI6N4NKOOOB3DCL7NW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Larry Hastings]

On 4/15/21 6:09 PM, Barry Warsaw wrote:

On Apr 15, 2021, at 17:47, Oscar Benjamin  wrote:

Would it be problematic to postpone making __future__.annotations the default?

This is a good question, and I think the SC would really like to know if 
anybody has objections to  postponing this to 3.11.  We haven’t discussed it 
yet (this whole topic is on our agenda for next Monday), but it might be the 
best thing to do given where we are in the 3.10 release cycle.  It would give 
everyone a chance to breathe and come up with the right long term solution.



I don't have any objections, and I certainly see the wisdom in such a 
decision.



Best wishes,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5B2SPKPJHNROKQORQIFNKUESNUUSPLT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Guido van Rossum]

On Thu, Apr 15, 2021 at 18:20 Barry Warsaw  wrote:

> On Apr 15, 2021, at 17:47, Oscar Benjamin 
> wrote:
> >
> > Would it be problematic to postpone making __future__.annotations the
> default?
>
> This is a good question, and I think the SC would really like to know if
> anybody has objections to  postponing this to 3.11.  We haven’t discussed
> it yet (this whole topic is on our agenda for next Monday), but it might be
> the best thing to do given where we are in the 3.10 release cycle.  It
> would give everyone a chance to breathe and come up with the right long
> term solution.


As long as you know that’s not zero cost either. Undoing the work will be
complex (merge conflicts will abound) and and all testing and
implementation people have done assuming PEP 563 semantics will still have
to be redone.

And note that there are some corner cases where the “PEP 563 semantics” in
3.10 could not be made fully compatible with the behavior under `from
__future__ Import annotations` in 3.9 (in truth, because not enough testing
was done under those semantics in 3.7-3.9).

But it could be done, and that’s the way PEP 649 is currently written
(using `from __future__ import co_annotations`).



-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ILL6UH2KS4NXL57LGEAUUA4MVNRK3OOB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Inada Naoki]

On Fri, Apr 16, 2021 at 9:49 AM Oscar Benjamin
 wrote:
>
>
> > That said, I agree it is better that this came up before the feature freeze 
> > than after the release. And I am willing to accept that the hypothetical 
> > future where annotations are not always syntactically expressions (which 
> > did not even exist before this week) is less important than backwards 
> > compatibility.
>
> Would it be problematic to postpone making __future__.annotations the default?
>

__future__.annotation is the default since 2020-10-04.
https://github.com/python/cpython/commit/044a1048ca93d466965afc027b91a5a9eb9ce23c#diff-ebc983d9f91e5bcf73500e377ac65e85863c4f77fd5b6b6caf4fcdf7c0f0b057

After that, many changes are made on compiler and other places.
So reverting the change is not so simple.

And personally, I love static typing but I don't use type hint for
performance/memory usage reason.
I spend much effort to optimize PEP 563 to minimize type hinting overhead.
So it's very sad that if I can not use type hinting when I can drop
Python 3.9 support.

So if PEP 649 is accepted, I want to use it since Python 3.10.
Otherwise, I can not use type hinting even after I dropped Python 3.9
support.

But it is up to release manager and steering council.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6ZYCD63KBP2EPDZIYJD2IPHCVRV4LQGP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Typing syntax and ecosystem

[Barry Warsaw]

On Apr 15, 2021, at 17:49, Luciano Ramalho  wrote:

> Anyone who uses a language with support for static typing expects the
> language distribution to include a type checker. Since that may be
> beyond our means, then the least we could do is have some official
> guidance on the matter, maybe in the form of a "Typing HOWTO" as part
> of the official docs, and linked from the "typing" module docs.

Actually, I think it’s time for a comprehensive guide to type annotations.  
Just anecdotally, I was trying to annotate a library of mine and was having an 
impossible time of it, until a chat with Guido lead me to @typing.overload.  
That solved my problem intuitively and easily, but I just didn’t know about it. 
 Right now, there’s information spread out all over the place, the stdlib 
documentation, tool documentation, StackOverflow :D etc.  It’s a complicated 
topic that I think a comprehensive guide, a tutorial, etc. could really help 
with.

One of my favorite frameworks for thinking about documentation on a topic such 
as this is:

https://documentation.divio.com/

I really think that would help people get into Python type annotations, both 
casually and deeply.

> I volunteer to help with a "Typing HOWTO". For the next few months, I
> can offer to review if someone else writes it. In the second semester,
> I could write it myself, if the experts on typing and the type
> checkers would be willing to review it.

I don’t know whether I’ll have time to *start* something any time soon, but I 
would also volunteer to be a reviewer and/or provide some content.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J56RVQHB76W5LY55OJ6NYS5KQXLKK7RJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Barry Warsaw]

On Apr 15, 2021, at 17:47, Oscar Benjamin  wrote:
> 
> Would it be problematic to postpone making __future__.annotations the default?

This is a good question, and I think the SC would really like to know if 
anybody has objections to  postponing this to 3.11.  We haven’t discussed it 
yet (this whole topic is on our agenda for next Monday), but it might be the 
best thing to do given where we are in the 3.10 release cycle.  It would give 
everyone a chance to breathe and come up with the right long term solution.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M5UZGWZRADZLG3ZF3QGSOFXWPSBBBPSY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Typing syntax and ecosystem

[Luciano Ramalho]

Today I stumbled upon another false positive on Mypy:
https://github.com/python/mypy/issues/4660 .
The issue is more than 3 years old, but it refers to a language
feature (__init_subclass__) which was added in Python 3.6, released in
late 2016.

I understand that maintaining a type checker is a lot of work, and
requires a particular set of skills, besides interest in the subject
matter.

So maybe the current state is the best we can offer.

But I do think we should mention this situation somewhere in the
official documentation.

Anyone who uses a language with support for static typing expects the
language distribution to include a type checker. Since that may be
beyond our means, then the least we could do is have some official
guidance on the matter, maybe in the form of a "Typing HOWTO" as part
of the official docs, and linked from the "typing" module docs.

Such a HOWTO could provide a gentle introduction to the topic, but
also cover the tools—which the existing docs and PEPs avoid. It should
also explain that the tools are external, as is typeshed, which means
there is always a lag between new language features and libraries and
the support provided by the tools.

I volunteer to help with a "Typing HOWTO". For the next few months, I
can offer to review if someone else writes it. In the second semester,
I could write it myself, if the experts on typing and the type
checkers would be willing to review it.

Cheers,

Luciano







On Wed, Apr 14, 2021 at 2:19 PM Brett Cannon  wrote:
>
>
>
> On Tue, Apr 13, 2021 at 5:12 PM Hugh Fisher  wrote:
>>
>> On Wed, 14 Apr 2021 at 08:21, Barry Warsaw  wrote:
>> > I wouldn’t necessarily be opposed to bundling a type checker with the 
>> > interpreter/stdlib, but I think there are some issues with it.  Just off 
>> > the top of my head (there are undoubtedly many more issues to resolve):
>> >
>> > * Which type checker would we adopt or adapt, if any?
>>
>> Mypy.
>> This has become an implementation issue, not one of which type
>> system to adopt. A lot of code, even in the stdlib, has been annotated
>> but I'm not aware of multiple different annotations with different
>> semantics or type systems being created.
>
>
> But there are feature concerns there as well, e.g. mypy and pytype offer 
> different "extras", even if the type checkers all align on semantics (which I 
> know they work on in the typing SIG). There's also variance in maintenance, 
> code complexity, etc. To me, this is not a clear-cut "mypy wins" situation.
>
> And I know Luciano said mypy because it's "the type checker hosted in the 
> python organization on github", but I don't know if the SC would approve that 
> today (Guido brought mypy into the org during his BDFL days), and instead my 
> guess is it would have ended up in the psf org like Black did.
>
>>
>>
>> For example, type equivalence by name only is used in Ada (or was,
>> it's been many years) and probably other languages. In equivalence
>> by name, the following code would not pass the type checker.
>> x : list[int]
>> y : list[int]
>> x = y # Type error
>>
>> But I'm not aware of anyone implementing type by name equivalence
>> for Python, and the original PEP 483 seems to explicitly close off that
>> possibility. Instead the assumption seems to be Java/C++ structural
>> equivalence for types.
>>
>> Skimming a bunch of current type system related PEPs, I'm not seeing
>> anything that a Java/C++ programmer would find unfamiliar. And this is
>> probably a good thing.
>>
>> > * Which parts of the typing system require more frequent release cycles?
>> > * Is there a core technology that could be put in the stdlib and still 
>> > allow experimentation?
>> > * Would the type checker authors become core developers?
>> > * Do the same feature release / deprecation policies apply?
>>
>> No answers from me.
>
>
> My guess is the closest we would ever come is some ensuretypechecker 
> situation like we have with ensurepip, but pip is done that way for 
> bootstrapping reasons and we don't need a type checker for bootstrapping.
>
> -Brett
>
>>
>>
>> > I would still be opposed to requiring type hinting in Python.
>>
>> I'm opposed to requiring type hints on everything, I want to still be
>> able to write
>> x = 1
>> x = "hello"
>> etc without declaring any kind of type for x.
>>
>> --
>>
>> cheers,
>> Hugh Fisher
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/OABB53NTCBU6EKYQVVVY4IU2275XO4R4/
>> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg

[Python-Dev] Re: In support of PEP 649

[Oscar Benjamin]

On Fri, 16 Apr 2021 at 01:13, Guido van Rossum  wrote:
>
> On Thu, Apr 15, 2021 at 16:48 Christopher Barker  wrote:
>>
>> And as I noted in my last post — many folks have not been paying attention 
>> to the typing discussions because they didn’t realize it concerned them.
>
> It seems a little disingenuous to claim discussions about annotations don’t 
> concern you when you’re actively using them (for typing, no less, in the case 
> of pydantic). And I am sure a project as popular (by their own description) 
> as pydantic will find a way forward if PEP 649 is rejected, despite 
> overdramatized claims.

It looks to me like pydantic were well aware that this affected them
and have been working over time to try and support this. They probably
also had their eye on PEP 649 or hoped that something else would come
along to fix this but they can now see the crunch point approaching.

As Chris says though other libraries/developers have been using
annotations for some time and might not have realised that typing
considerations could undermine their other uses for annotations since
annotations predate typing and typing is an optional feature.

> That said, I agree it is better that this came up before the feature freeze 
> than after the release. And I am willing to accept that the hypothetical 
> future where annotations are not always syntactically expressions (which did 
> not even exist before this week) is less important than backwards 
> compatibility.

Would it be problematic to postpone making __future__.annotations the default?


Oscar
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VCFHDQFOBQXZDKKPSTU5FNKP2AZIJMX2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Guido van Rossum]

On Thu, Apr 15, 2021 at 16:48 Christopher Barker 
wrote:

> And as I noted in my last post — many folks have not been paying attention
> to the typing discussions because they didn’t realize it concerned them.
>

It seems a little disingenuous to claim discussions about annotations don’t
concern you when you’re actively using them (for typing, no less, in the
case of pydantic). And I am sure a project as popular (by their own
description) as pydantic will find a way forward if PEP 649 is rejected,
despite overdramatized claims.

That said, I agree it is better that this came up before the feature freeze
than after the release. And I am willing to accept that the hypothetical
future where annotations are not always syntactically expressions (which
did not even exist before this week) is less important than backwards
compatibility.


-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZGRV3QHTKPFHMJY6R4Z7J7G3LPDTRA4O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]

[Irit Katriel via Python-Dev]

Thomas and the rest of the SC,

Thank you for updating us and for your kind words about the PEP.

We agree that it is safer to include this in 3.11 and not rush it into
3.10. As you say, having time to develop the integration with asyncio
before finalizing the API will give us more confidence in the design.

We would like to commit the code into the 3.11 development branch soon, as
you suggest.  We will be looking forward to the (provisional?) acceptance
of the PEP by the SC so that we can do this.

Best regards
Irit, Yury & Guido

On Thu, Apr 15, 2021 at 6:48 PM Thomas Wouters  wrote:

>
> Irit, Yury and Guido,
>
> Thanks for submitting PEP 654 (Exception Groups and except *). The
> Steering Council isn’t done debating this PEP, but we wanted to give an
> update so everyone knows what to expect.
>
> We are mostly in favour of the PEP, but we’re worried about the timing --
> not just because there is still some discussion going on, but also the time
> available for us to discuss the proposal, and to validate the new API once
> it’s accepted. We are rapidly approaching  feature freeze for Python 3.10,
> and while we’re not particularly worried about the implementation of this
> PEP, we would like to see actual usage of the new API with enough time to
> tweak it before the official release. That is to say, we would like to see
> the "analysis of how exception groups will likely be used in asyncio
> programs" turn into "here is how exception groups are used in asyncio
> programs" before the final release, like in the asyncio module and the
> standard library (for example, with TaskGroups). There’s also a
> presentation scheduled in the PyCon US Language Summit, which may lead to
> more discussion of the PEP and perhaps evolution of its ideas.
>
> What would you and others feel about postponing the PEP until Python 3.11,
> getting it in as soon as the branch opens, so that potential users of the
> API -- asyncio or otherwise -- can provide us with practical experience to
> validate some of the assumptions made in the proposal, and to discover any
> warts or pitfalls we missed or misjudged? I don’t mean that as a slight on
> the PEP, which is thoughtful, thorough and excellently written, but as a
> practical observation of the complexity involved. Most of the PEP is more
> of a building block for other APIs than something many users will be
> directly exposed to, and we want to make sure the end result isn’t affected
> by unanticipated effects of these design decisions. (This would potentially
> also give third-party users more time to evaluate the API, although we know
> adoption of alpha releases isn’t big enough to make this a very strong
> argument for waiting.)
>
> Like I said, we’re still discussing the PEP, so this hasn’t been decided.
> If you or others feel strongly that the benefit of postponing is too small
> (or the cost too high), we’ll take that into consideration. The SC has been
> somewhat busy of late, and we don’t want to let our schedule be the
> deciding factor here, but this may come down to there just not being enough
> time before the feature freeze to make a thoughtful decision.
>
> For the SC,
> Thomas.
> --
> Thomas Wouters 
>
> Hi! I'm an email virus! Think twice before sending your email to help me
> spread!
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WSXKY6FVJ7EFQ533OJE67ZIISAAX2QJE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Christopher Barker]

On Thu, Apr 15, 2021 at 4:20 PM Brett Cannon  wrote:

>
>  Making this an "us versus them" discussion just makes the whole situation
> feel confrontational when instead everyone is trying to figure out the best
> thing for everybody when there's no perfect answer.
>

I agree that that was strong language, but from the perspective of folks
not interested in Static Typing, there has been a lot of activity, or at
least discussion,  that makes it seem like this optional feature is driving
the evolution of the language.

So folks are going to get concerned, and maybe upset, if it looks like
changes might be made that may break other features in order to make static
typing work better.

And as I noted in my last post — many folks have not been paying attention
to the typing discussions because they didn’t realize it concerned them.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O57G2ZDD2VW7OB5IMPKACL775HBDSNIP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Paul Ganssle]

It seems evident that the problems with PEP 563 have been well-known at
least to pydantic for several years, as you can see in the issue Samuel
Colvin linked: https://github.com/samuelcolvin/pydantic/issues/2678

That said, while I do think that "please contact upstream when you see a
problem developing, not just before a major deadline" is a good lesson
to take away from this, it is probably not worth litigating the question
of the particular manner of the notification. As much as I think it
would have been good for this discussion to happen 6 months, 1 year, 2
years or 3 years ago, "before the code freeze" is a heck of a lot better
than "after the release" (which is often when the notification comes in,
and why we have been encouraging people to test against alphas and betas).

Hopefully we can let it rest by saying that the timing of learning about
this apparently urgent situation could have been much better, but it
could have been worse as well.

Best,
Paul

On 4/15/21 7:12 PM, Christopher Barker wrote:
> On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor  > wrote:
> > I'm a bit surprised that this topic is brought up just days before
> the feature freeze of Python 3.10.
>
> I have not idea about the technical details, but I think there is a
> bit of a discomment in the community:
>
> Annotations have been around for years.
>
> More recently, there was an official specification of how to use them
> for type hinting (was that PEP 484? a bit confused as that appears to
> still be provisional -- but anyway)
>
> Since the specification for type hinting, a lot of work has been done
> on/with static type checkers (e.g. MyPy).
>
> That work has informed the changes / improvements / proposals to the
> annotation and type system in Python.
>
> However, those of us that are not actively involved (and maybe not
> that interested) in static typing have been not paying a great deal of
> attention to those efforts.
>
> But some of us may, in fact, be using annotations for other reasons,
> but since we haven't been involved in the discussion, some issues may
> have been missed.
>
> -CHB
>
>
> -- 
> Christopher Barker, PhD (Chris)
>
> Python Language Consulting
>   - Teaching
>   - Scientific Software Development
>   - Desktop GUI and Web Development
>   - wxPython, numpy, scipy, Cython
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/5TGKKRZXGQROQKS2WX6WFGCHTMNUJYBF/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WETOB43S2H6RGVX7RKWOZNHZZHH245V6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Brett Cannon]

On Thu, Apr 15, 2021 at 12:01 PM Samuel Colvin  wrote:

> I've read the recent discussions
> 
> regarding PEP 649 and PEP 563 with interest, Larry Hastings recently
> contacted me when prompted
> 
> to do so in a related discussion.
>
> I maintain pydantic  which uses
> type annotations to provide runtime data validation.
>
> I'm extremely worried that if PEP 649 gets rejected, pydantic will not be
> able to fully support python 3.10 and may even end up getting abandoned, at
> best it will be much slower and more brittle.
>
> Please, please put pragmatism over principle and accept PEP 649.
>

Please don't phrase the decision in these terms. While for Pydantic PEP 649
is more pragmatic, that does not mean PEP 563 isn't pragmatic for other
people for other reasons. Making this an "us versus them" discussion just
makes the whole situation feel confrontational when instead everyone is
trying to figure out the best thing for everybody when there's no perfect
answer.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QAH7UBRM2PF3FEB7K4EYDDXJDD4G4CFQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Brett Cannon]

On Thu, Apr 15, 2021 at 10:53 AM Antoine Pitrou  wrote:

> On Thu, 15 Apr 2021 10:28:53 -0700
> Brett Cannon  wrote:
> > On Thu, Apr 15, 2021 at 8:50 AM Christopher Barker 
> > wrote:
> >
> > > On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner 
> > > wrote:
> > >
> > >> Paul Bryan:
> > >> > Seems like this is something that should make its way into stdlib?
> > >>
> > >> In the last 10 years, the trend is more to remove anything related to
> > >> packaging *outside* the stdlib :-)
> > >
> > >
> > > Well, maybe all of us don't think that's a good idea ;-)
> > >
> >
> > As an active participant in the PyPA side of the packaging ecosystem and
> > one of the maintainers of 'packaging', I am definitely NOT in favour of
> > moving any of 'packaging' into the stdlib, nor breaking it up. We are
> > taking distutils out of the stdlib for a reason. Backsliding on plans
> that
> > have been in place for years is not something I would want to see happen
> as
> > the motivations have not changed.
>
> This seems gratuitously dogmatic. Version numbers are a simple feature
> that is stably encoded in PEP 440. It's purely computational, does
> not involve reading or saving any persistent state in the filesystem,
> making any network access, etc. Why wouldn't it have its place in the
> stdlib?
>

Technically PyPA specs might start as PEPs but their official home is
packaging.python.org and updates can occur there outside of the PEP
process. In the case of version specifiers (
https://packaging.python.org/specifications/version-specifiers/), they
point back to the original PEP, but that is not universally true nor
guaranteed to hold in the future. In other words I personally don't view
the fact that it's a PEP as a reason to have it in the stdlib (else pretty
much every single packaging PEP is missing).

We also have other things standardized in PEPs and not covered in the
stdlib, e.g. @ as an operator.

So I don't think version comparison occurs often enough to be in the
stdlib, and the fact that an external project exists which isn't interested
in being put in the stdlib suggests to me it isn't worth doing.

But that's a separate topic to discuss at the language summit. :)


>
> > > But anyway, I would say removing anything *related* to packaging
> outside
> > > the stdlib is a bad idea -- requiring an external tool to build a
> package
> > > is OK, but requireing an external use packages, not so much.
> > >
> >
> > You can totally use a package already as things exist today and will in
> the
> > future; you're just wanting a specific way to interpret a package's
> > metadata which I don't view as the same thing.
>
> That specific way is a Python standard (PEP 440). Having the
> functionality in the stdlib would encourage people to use it. Not
> having it in the stdlib encourages people to use adhoc version parsing,
> or worse, naive string comparison.
>

I don't know if I draw the same line since the packaging community is
relying on 'packaging' without issue for this exact thing.


>
> > So asking every Python project to set a
> > __version__ isn't going to change accessibility of version numbers when
> it
> > comes to installed projects.
>
> That doesn't have much to do with the suggestion of shipping a Version
> class in the stdlib, though. Many projects already provide a
> __version__, and that field is often inspected programmatically in
> dependent packages (to work around behaviour changes, for example).
>

But this thread is about standardizing on __version__ which then led to
wanting to bring in some Version object into the stdlib to make that more
useful. So to me they are tied together, else a separate thread should
probably be started if a new module *just *for version parsing support is
desired.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AYJ4JBTIVQTJY7IQYZMYCUY6VSXSMXXY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Christopher Barker]

On Thu, Apr 15, 2021 at 3:29 PM Bernat Gabor  wrote:
> I'm a bit surprised that this topic is brought up just days before the
feature freeze of Python 3.10.

I have not idea about the technical details, but I think there is a bit of
a discomment in the community:

Annotations have been around for years.

More recently, there was an official specification of how to use them for
type hinting (was that PEP 484? a bit confused as that appears to still be
provisional -- but anyway)

Since the specification for type hinting, a lot of work has been done
on/with static type checkers (e.g. MyPy).

That work has informed the changes / improvements / proposals to the
annotation and type system in Python.

However, those of us that are not actively involved (and maybe not that
interested) in static typing have been not paying a great deal of attention
to those efforts.

But some of us may, in fact, be using annotations for other reasons, but
since we haven't been involved in the discussion, some issues may have been
missed.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5TGKKRZXGQROQKS2WX6WFGCHTMNUJYBF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Larry Hastings]

On 4/15/21 2:02 PM, Sebastián Ramírez wrote:

## Questions

I'm not very familiar with the internals of Python, and I'm not sure how the new syntax for 
`Union`s using the vertical bar character ("pipe", "|") work.

But would PEP 649 still support things like this?:

def run(arg: int | str = 0): pass

And would it be inspectable at runtime?



As far as I can tell, absolutely PEP 649 would support this feature.  
Under the covers, all PEP 649 is really doing is changing the 
destination that annotation expressions get compiled to.  So anything 
that works in an annotation with "stock" semantics would work fine with 
PEP 649 semantics too, barring the exceptions specifically listed in the 
PEP (e.g. annotations defined in conditionals, walrus operator, etc).



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RD4X7EGVTY6PIT76TWQCSWC6PLHMIL6Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Bernat Gabor]

Hello,

I'm a bit surprised that this topic is brought up just days before the
feature freeze of Python 3.10.  How did tools work until now with from
future import annotations? This feature has been in the language for years
(since 3.7), and I haven't heard until today anyone complaining about this
feature gap. Did everyone just told users not to use lazy evaluation of
type hints until now? Also there's been advertised for at least a year that
next python will make it default 樂

Thanks,

Bernat

On Thu, Apr 15, 2021, 23:13 Paul Bryan  wrote:

> I've been working on a similar framework, Fondat [1], that shares similar
> objectives as Pydantic and FastAPI, albeit implementing in a different
> manner.
>
> I believe other frameworks will have to "run a gauntlet" (as I already
> have)  to deal with the affects of PEP 563. If anyone on these teams are
> interested in how I've dealt with such issues, please feel free to contact
> me directly.
>
> I also recommend members of these teams monitor and/or participate in the
> typing-sig (and maybe the python-dev) mailing list, as discussions in these
> lists are likely to affect their work in the future.
>
> Paul
>
> [1] https://github.com/fondat/fondat-core
>
>
> On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:
>
> I should point out that "accept PEP 649" and "break pydantic" are not the
> only options here. The thing that will break pydantic is the end of PEP
> 563's deprecation period, not a failure to implement PEP 649.
>
> Other viable options include:
>
> - Leave PEP 563 opt-in until we can agree on a solution to the problem.
> - Leave PEP 563 opt-in forever.
> - Deprecate PEP 563 and go back to status quo ante.
>
> I haven't followed this closely enough — if PEP 649 were accepted today,
> would it even be ready for use before the 3.10 code freeze (which is in a
> few weeks)?
>
> Assuming this is a real problem (and based in part on how long it took
> for attrs to get what support it has for PEP 563
> , I wouldn't be
> surprised if PEP 563 is quietly throwing a spanner in the works in several
> other places as well), my vote is to leave PEP 563 opt-in until at least
> 3.11 rather than try to rush through a discussion on and implementation of
> PEP 649.
>
> Best,
> Paul
> On 4/15/21 4:20 PM, Bluenix wrote:
>
> Please accept PEP 649!
>
>
> Python's type hinting has become so much more useful than originally thought, 
> and without this change much of that will be hindered. For example (you 
> already know about Pydantic and FastAPI) 
> [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows 
> you to use typehinting to specify how arguments should be converted. Take the 
> following code:
>
>
> ```py
>
> import discord
>
> from discord.ext import commands
>
>
> bot = commands.Bot(command_prefix='>')
>
>
> @bot.command()
>
> # discord.py reads the typehints and converts the arguments accordingly
>
> async def reply(ctx, member: discord.Member, *, text: str):  # ctx is always 
> passed
>
> await ctx.send(f'{member.mention}! {text}')
>
>
> bot.run('token')
>
> ```
>
>
> I must say, this is extremely ergonomic design! Don't break it :)
>
> ___
>
> Python-Dev mailing list -- python-dev@python.org
>
> To unsubscribe send an email to python-dev-le...@python.org
>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
>
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
>
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PBGFXPC7D3NFLJSFQBDAQQCW6JCMVCFC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7KU4WMWD3U6AFVDFOGVZER4QWZJFPAB4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Larry Hastings]

On 4/15/21 2:49 PM, Paul Ganssle wrote:
I haven't followed this closely enough — if PEP 649 were accepted 
today, would it even be ready for use before the 3.10 code freeze 
(which is in a few weeks)?



Perhaps I'm a poor judge of the quality of my own code.   But I'd say I 
have a working prototype and it seems... fine.  It didn't require a lot 
of changes, it just needed the right changes in the right spots.  
Getting my interactions with symtable and compile were the hardest parts 
and I think those are all sorted out now.



Cheers,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6VFTVY7CAXMYHC4GRA4NV2LTUFW6ESAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Paul Bryan]

I've been working on a similar framework, Fondat [1], that shares
similar objectives as Pydantic and FastAPI, albeit implementing in a
different manner. 

I believe other frameworks will have to "run a gauntlet" (as I already
have)  to deal with the affects of PEP 563. If anyone on these teams
are interested in how I've dealt with such issues, please feel free to
contact me directly.

I also recommend members of these teams monitor and/or participate in
the typing-sig (and maybe the python-dev) mailing list, as discussions
in these lists are likely to affect their work in the future.

Paul 

[1] https://github.com/fondat/fondat-core


On Thu, 2021-04-15 at 17:49 -0400, Paul Ganssle wrote:
> I should point out that "accept PEP 649" and "break pydantic" are not
> the only options here. The thing that will break pydantic is the end
> of PEP 563's deprecation period, not a failure to implement PEP 649.
> Other viable options include:
> - Leave PEP 563 opt-in until we can agree on a solution to the
> problem.
> - Leave PEP 563 opt-in forever.
> - Deprecate PEP 563 and go back to status quo ante.
> 
> I haven't followed this closely enough — if PEP 649 were accepted
> today, would it even be ready for use before the 3.10 code freeze
> (which is in a few weeks)?
> 
> Assuming this is a real problem (and based in part on how long it
> took for attrs to get what support it has for PEP 563, I wouldn't be
> surprised if PEP 563 is quietly throwing a spanner in the works in
> several other places as well), my vote is to leave PEP 563 opt-in
> until at least 3.11 rather than try to rush through a discussion on
> and implementation of PEP 649.
> 
> Best,
> Paul
> On 4/15/21 4:20 PM, Bluenix wrote:
> 
> > Please accept PEP 649!
> > 
> > Python's type hinting has become so much more useful than originally 
> > thought, and without this change much of that will be hindered. For example 
> > (you already know about Pydantic and FastAPI) 
> > [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows 
> > you to use typehinting to specify how arguments should be converted. Take 
> > the following code:
> > 
> > ```py
> > import discord
> > from discord.ext import commands
> > 
> > bot = commands.Bot(command_prefix='>')
> > 
> > @bot.command()
> > # discord.py reads the typehints and converts the arguments accordingly
> > async def reply(ctx, member: discord.Member, *, text: str):  # ctx is 
> > always passed
> > await ctx.send(f'{member.mention}! {text}')
> > 
> > bot.run('token')
> > ```
> > 
> > I must say, this is extremely ergonomic design! Don't break it :)
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PBGFXPC7D3NFLJSFQBDAQQCW6JCMVCFC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Simon Cross]

Replying to myself to update my own thoughts:

I think __version__ and packaging tools answer separate questions.

__version__ answers the question of "this module I imported, what
version is it".

The packaging tools answer the question "what dependency does this
distribution satisfy".

This mismatch is why importlib.metadata.version("bs4") doesn't return
a useful answer. A single distribution might even install multiple
top-level modules with different versions.

Schiavo,
Simon
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O242SRB6SBLBBMPCOLCKURBEXXHUSJZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Paul Ganssle]

I should point out that "accept PEP 649" and "break pydantic" are not
the only options here. The thing that will break pydantic is the end of
PEP 563's deprecation period, not a failure to implement PEP 649.

Other viable options include:

- Leave PEP 563 opt-in until we can agree on a solution to the problem.
- Leave PEP 563 opt-in forever.
- Deprecate PEP 563 and go back to status quo ante.

I haven't followed this closely enough — if PEP 649 were accepted today,
would it even be ready for use before the 3.10 code freeze (which is in
a few weeks)?

Assuming this is a real problem (and based in part on how long it took
for attrs to get what support it has for PEP 563
, I wouldn't be
surprised if PEP 563 is quietly throwing a spanner in the works in
several other places as well), my vote is to leave PEP 563 opt-in until
at least 3.11 rather than try to rush through a discussion on and
implementation of PEP 649.

Best,
Paul

On 4/15/21 4:20 PM, Bluenix wrote:
> Please accept PEP 649!
>
> Python's type hinting has become so much more useful than originally thought, 
> and without this change much of that will be hindered. For example (you 
> already know about Pydantic and FastAPI) 
> [discord.py](https://github.com/Rapptz/discord.py)'s commands system allows 
> you to use typehinting to specify how arguments should be converted. Take the 
> following code:
>
> ```py
> import discord
> from discord.ext import commands
>
> bot = commands.Bot(command_prefix='>')
>
> @bot.command()
> # discord.py reads the typehints and converts the arguments accordingly
> async def reply(ctx, member: discord.Member, *, text: str):  # ctx is always 
> passed
> await ctx.send(f'{member.mention}! {text}')
>
> bot.run('token')
> ```
>
> I must say, this is extremely ergonomic design! Don't break it :)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GT2HQDH2HOZFSOTA5LHTFL5OV46UPKTB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Paul Moore]

On Thu, 15 Apr 2021 at 21:03, Christopher Barker  wrote:

> I am leaning toward at least SOME PEP about this -- having  __version__ 
> around as a semi-convention with no official recommendation as to if or how 
> it should be used is kind of a mess.

On the other hand, having a PEP that says you "may" have a version
attribute, but mandates nothing, is pretty useless. People who conform
to that convention don't need telling, people who don't have little
reason to change, and people wanting to use the information still have
to cater for it being missing, so they gain nothing.

But a PEP that makes __version__ a requirement fails instantly,
because there's too many packages in the wild that violate it.

Welcome to the world of packaging PEPs :-)

> I'm not sure what to make of all this, though I'm leaning toward better 
> suporting the distiction by asking for __version__ strings in top-level 
> packages -- and maybe making importlib.metadata.version a bit smarter about 
> looking for packages, and not just distributions.

Personally, I'm leaning more and more towards the view that it's
*distributions* that are versioned, and that's an established
practice. Versioning packages as well (except in the very limited
sense that package X installed by distribution Y can expose Y's
version via its API) only adds confusion and makes the situation less,
not more, useful for users.

So I'm now -1 on any PEP to try to formalise the use of __version__.
If people want to discuss the use of a __version__ attribute, they
should consider contributing to the discussions on versions in
https://packaging.python.org/guides/ (where the necessary context is
already present).

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ECJ2HHV2EXJ7GOJU7ILLQVVLF54VVD3U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[David Mertz]

On Thu, Apr 15, 2021 at 9:54 PM Christopher Barker 
wrote:

> Or rather, the below is what I would find really nice to be able to do.
>
>> ver = robust_version(module)
>> if ver >= (5, 2, 1):
>> doit_modern_style()
>> elif ver < (5, 2, 1):
>> doit_old_style
>> else:
>> doit_unversioned_style()
>>
>
> Exactly -- and I htink we are close, if pacakges(modules) had compliant
> __version__ strings, then you could do
> (with the Version object from packaging -- maybe why it should be in the
> stdlib)
>
> ver = Version(module.__version__)
> if ver >= Version("5.2.1"):
> doit_modern_style()
> elif ver < Version("5.2.1"):
> doit_old_style
> else:
> doit_unversioned_style()
>

This wouldn't be bad.  A little more verbose, but fine.

The thing is, I almost never care about the version of a *distribution* in
my programs. I care about a version of the *package*.  Since, as Chris
mentions, a distribution could contain multiple packages, the same package
could be vendorized (in different versions) by different distributions.
The question I want to ask above is always because "version such-and-such
changed an API," and that's about a package.  When I care about
distribution versions, it is because there are dependencies stated in
requirements.txt or environment.yaml or whatever.deb.

Asking for the `.__version__` attributes feels like needless redundancy in
Chris' variation.  The constructor should be able to ask "Is this a module?
If so, does it have that attribute".  But whatever, I can learn to do that
easily enough.

Someone else mentioned that the versions  of standard library modules are
an odd collection of relics.  I do not disagree, of course.  But sometimes
the stdlib modules *do* change.  I thought of the `re` example because I
was doing a tutorial on it, and something or another depended on a change
between 3.8 and 3.9 (I think, I forget the detail).  For training material,
this can just be approached as describing the Python version... but for
running code, a version check could be relevant.

Therefore, I think this `Version` constructor (or `robust_version()`
function, or whatever) should have access to that.  I.e. if not an
automated update to `.__version__` inside each stdlib module during builds,
then a fallback to "If this is a stdlib module, report sys.version_info
(within the right kind of "Version" object).


-- 
The dead increasingly dominate and strangle both the living and the
not-yet born.  Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZIGLNTQC7WT43UMLQQHT7UB57KX7EFCX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Sebastián Ramírez]

Thanks Brett Cannon for suggesting to get Samuel Colvin (Pydantic) and me, 
Sebastián Ramírez (FastAPI and Typer) involved in this.

TL;DR: it seems to me PEP 649 would be incredibly important/useful for 
Pydantic, FastAPI, Typer, and similar tools, and their communities.

## About FastAPI, Pydantic, Typer

Some of you probably don't know what these tools are, so, in short, FastAPI is 
a web API framework based on Pydantic. It has had massive growth and adoption, 
and very positive feedback.

FastAPI was included for the first time in the last Python developers survey 
and it's already the third most used web framework, and apparently the fastest 
growing one: https://www.jetbrains.com/lp/python-developers-survey-2020/.

It was also recently recommended by ThoughtWorks for the enterprises: 
https://www.thoughtworks.com/radar/languages-and-frameworks?blipid=202104087

And it's currently used in lots of organizations, some widely known, chances 
are your orgs already use it in some way.

Pydantic, in very short, is a library that looks a lot like dataclasses (and 
also supports them), but it uses the same type annotations not only for type 
hints, but also for data validation, serialization (e.g. to JSON) and 
documentation (JSON Schema).

The key feature of FastAPI (thanks to Pydantic) is using the same type 
annotations for _more_ than just type hinting: data validation, serialization, 
and documentation. All those features are provided by default when building an 
API with FastAPI and Pydantic.

Typer is a library for building CLIs, based on Click, but using the same ideas 
from type annotations, from FastAPI and Pydantic.

## Why PEP 649

You can read Samuel's message to this mailing list here: 
https://mail.python.org/archives/list/python-dev@python.org/thread/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/

And a longer discussion of how PEP 563 affects Pydantic here: 
https://github.com/samuelcolvin/pydantic/issues/2678

He has done most of the work to support these additional features from type 
annotations. So he would have the deepest insight into the tradeoffs/issues.

>From my point of view, just being able to use local variables in Pydantic 
>models would be enough to justify PEP 649. With PEP 563, if a developer 
>decides to create a Pydantic model inside a function (say a factory function) 
>they would probably get an error. And it would probably not be obvious that 
>they have to declare the model in the top level of the module.

The main feature of FastAPI and Pydantic is that they are very easy/intuitive 
to use. People from many backgrounds are now quickly and efficiently building 
APIs with best practices.

I've read some isolated comments of people that were against type annotations 
in general, saying that these tools justify adopting them. And I've also seen 
comments from people coming from other languages and fields, and adopting 
Python just to be able to use these tools.

Many of these developers are not Python experts, and supporting them and their 
intuition as much as possible when using these tools would help towards the PSF 
goal to:

> [...] support and facilitate the growth of a diverse and international 
> community of Python programmers.

## Community support

To avoid asking people to spam here, Samuel and I are collecting "likes" in:

* This tweet: https://twitter.com/tiangolo/status/1382800928982642692
* This issue: https://github.com/samuelcolvin/pydantic/issues/2678

I just sent that tweet, I expect/hope it will collect some likes in support by 
the time you see it.

## Questions

I'm not very familiar with the internals of Python, and I'm not sure how the 
new syntax for `Union`s using the vertical bar character ("pipe", "|") work.

But would PEP 649 still support things like this?:

def run(arg: int | str = 0): pass

And would it be inspectable at runtime?

## Additional comments

The original author of PEP 563 was Łukasz Langa.

I was recently chatting with him about Typer and annotations. And he showed 
interest, support, and help.

I think he probably knows the benefits of the way these libraries use type 
annotations and I would like/hope to read his opinion on all this.

Or alternatively, any possible ideas for how to handle these things in tools 
like Pydantic.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J7YKCCCV2GL7HUDKG7DX4INAT65SXYRF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Bluenix]

Please accept PEP 649!

Python's type hinting has become so much more useful than originally thought, 
and without this change much of that will be hindered. For example (you already 
know about Pydantic and FastAPI) 
[discord.py](https://github.com/Rapptz/discord.py)'s commands system allows you 
to use typehinting to specify how arguments should be converted. Take the 
following code:

```py
import discord
from discord.ext import commands

bot = commands.Bot(command_prefix='>')

@bot.command()
# discord.py reads the typehints and converts the arguments accordingly
async def reply(ctx, member: discord.Member, *, text: str):  # ctx is always 
passed
await ctx.send(f'{member.mention}! {text}')

bot.run('token')
```

I must say, this is extremely ergonomic design! Don't break it :)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S2O7SE4QZQARAYSCOT7PQUEPNENHDJTQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Thu, Apr 15, 2021 at 9:36 AM David Mertz  wrote:

> I was so hopeful about this, but in the end... not really.  I have not
> used this capability before.  Here are a few different situations I know of:
>

...

...
re PackageNotFoundError('re')
statistics PackageNotFoundError('statistics')
pandas 1.2.4
vaex 4.1.0
bs4 PackageNotFoundError('bs4')

>
> It seems like (somehow or another) importlib.metadata is doing something
> perhaps more useful for vaex.  But it is doing distinctly worse for re,
> statistics, and bs4.
>

funny you should try bs4, which I also used as an example in another post.

But what you have found is that there is a difference between a pacakge
(which you can import) and a "distribution" which is something you install.

importlib.metadata is looking for distributions.

re and statistics are not distributions, but rather, built in pacakges
(modules).

bs4 is not a distributiuion, it is a package that is provided by the
"beautifulsoup4" distribution.

In [3]: importlib.metadata.version("beautifulsoup4")
Out[3]: '4.9.3'

Frankly, I'm still a bit confused about the distiction, but I do know for
sure that a single distribution (maybe somethign that you can install via
PyPi) can install more than one top-level package -- and certainly the
packages/modules installed can have a different name than the distribution
name.

And if a distribution installs more than one package, they may have
different version numbers.

I'm not sure what to make of all this, though I'm leaning toward better
suporting the distiction by asking for __version__ strings in top-level
packages -- and maybe making importlib.metadata.version a bit smarter about
looking for packages, and not just distributions.

If you look at the docstring of metadata.version:

"""
In [2]: importlib.metadata.version?
Signature: importlib.metadata.version(distribution_name)
Docstring:
Get the version string for the named package.

:param distribution_name: The name of the distribution package to query.
:return: The version string for the package as defined in the package's
"Version" metadata key.
File:  ~/miniconda3/envs/py3/lib/python3.9/importlib/metadata.py
Type:  function
"""

It's a bit inconsistent with the use of the term "distribution" vs
"package". That should get cleaned up if nothing else.

Also, the Exception raised is "PackageNotFoundError" -- which should maybe
be "DistributionNotFoundError"?


Version is arguably useful from the package user side. As I believe Victor
mentioned, there are two uses for version information: display to the user
-- for which version strings are fine, or programmatic comparison -- for
which something like the Version object is very helpful. Do we only need to
use version information programmatically when we are creating (or
installing) packages? I don't think so -- I know I have code that (poorly)
does version checking programmatically.


> Or rather, the below is what I would find really nice to be able to do.
>
> ver = robust_version(module)
> if ver >= (5, 2, 1):
> doit_modern_style()
> elif ver < (5, 2, 1):
> doit_old_style
> else:
> doit_unversioned_style()
>

Exactly -- and I htink we are close, if pacakges(modules) had compliant
__version__ strings, thenyou could do
(with the Version object from packaging -- maybe why it should be in the
stdlib)

ver = Version(module.__version__)
if ver >= Version("5.2.1"):
doit_modern_style()
elif ver < Version("5.2.1"):
doit_old_style
else:
doit_unversioned_style()

And if my PR is accepted (which looks unlikley) to allow camparison between
Version objects and strings:

ver = Version(module.__version__)
if ver >= "5.2.1":
doit_modern_style()
elif ver < "5.2.1":
doit_old_style
else:
doit_unversioned_style()

A key distiction here from the importlib.metadata approach is the level of
indirection -- I like being able to ask a module itself what version it is,
rather than asking some other part of teh system to go look it up for me.
So I could do:

import numpy as np

print("using np version:", np.__version)

And this is pretty consitent with the rest of Python, where many objects
(functions, classes, modules) have a __name__ attribute if things "know"
their name, shouldn't they "know" their version as well?

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQYKOZ273ISH3KLWTDGNRTSYUCQGYD33/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[redradist]

@Christian Heimes

RAII is more than cleaning resource at the end of function, it is that while 
you are using class - resource will be available ...
You can move resource using modern C++ `std::move` ant lots of other features 
that could improve readability and maintainability of the code
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NUKSLDAZ5FIZUE7NXZIGRJ64PTKJMKTQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Christopher Barker]

I haven't commented on these, as I don't use type annotations in the
"usual" way.,
but:

On Thu, Apr 15, 2021 at 12:06 PM Samuel Colvin  wrote:

> I maintain pydantic  which uses
> type annotations to provide runtime data validation.
>

I maintain in in-house system kinda similar to pydantic that doesn't use
type annotations for validation, but does use them for other things (mostly
serializing/deserializing to/from JSON), and other kinds of data validation.

> using `typing.get_type_hints()` is not a good replacement for type
annotations that are accessible as python objects.

Absolutely. It is very, very, handy to simply have the object itself easily
accessible.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W7KPZQFMJ2WVW26KKLQDAXHWWTTWFYKQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Thu, Apr 15, 2021 at 12:27 PM Terry Reedy  wrote:

> At least some in the stdlib have not been.  For re.__version__, not
> since Nov 2001.
>

>From Serhiy over on python-ideas, he's started an effort to clean that up.

https://mail.python.org/archives/list/python-dev@python.org/thread/KBU4EU2JULXSMUZULD5HJJWCGOMN52MK/
).

I believe there is an unfinished PR

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRKC3EJWJZFFAFH7VP7W2GCIRFFEIEDK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Thu, Apr 15, 2021 at 9:38 AM Barry Warsaw  wrote:

> From a library maintainers point of view, I personally want to get away
> from using __version__ strings at all.  They’re kind of a pain to remember
> to bump on every new release.


That’s a tooling issue, and I don’t think that Python itself should take
any official stance on the tooling. But yes, anything that Python takes an
official stance on should be able to be easily supported with
not-too-complex tools.and I think a __version__ string fits this need.

Personally, I use the __version__ string in my packages as the canonical
source when I bold my packages.

And there are fancier tools for more complex needs.

> PEP 396 has been dormant for a long time and I have no interest in
pushing it forward any more.  If someone else wants to take up the cause, I
recommend creating a new PEP and referring back to 396.

Fair enough -- I'm still gathering data to see if I should do just that.

I am leaning toward at least SOME PEP about this -- having  __version__
around as a semi-convention with no official recommendation as to if or how
it should be used is kind of a mess.

-CHB
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FXIM52YHGECNJMKPA4E6UXJNRMZ2CUO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Terry Reedy]

On 4/15/2021 12:35 PM, David Mertz wrote:


re 2.2.1


re.__version__ was last modified 2001-10-07 by F. Lundh in 
bec95b9d8825b39cff46a8c645fa0eeb8409854e.  What does '2.2.1' mean?  Only 
Fredrik knows.  The commit message says that he rewrote the pattern.sub 
and pattern.subn methods in C.  That should have been user invisible 
except for speed.


I believe that it is dead data left only for back compatibility and 
because of an inability to give a deprecation warning.



re PackageNotFoundError('re')


It seems like (somehow or another) importlib.metadata is doing something 
perhaps more useful for vaex.  But it is doing distinctly worse for re, 


Is an effectively meaningless and misleading value better than none?

IDLE once had a version last set, I believe, in 2.5 or 2.6, by K. 
Kaiser, with whatever meaning it had to him.  I removed it in 3.6 
(allowed by PEP 434 + RM decision). No one complained.


The real version of any stdlib module is the Python version it is 
released with, plus and external app versions it is associated with 
(sqlite and ssl, for instance).  It has been suggested, if not decided, 
that other versions should go.



--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4AA7I3RS6D3UDIBZ6ELTEEWXKXIGE5WS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Terry Reedy]

On 4/15/2021 12:38 PM, Barry Warsaw wrote:

On Apr 14, 2021, at 23:11, Christopher Barker  wrote:


You wrote the original PEP, so of course you can withdraw it (or reject it), 
but...

Are you sure? See this discussion, I don't think it's as simple as all 

that.


 From a library maintainers point of view, I personally want to get away from 
using __version__ strings at all.  They’re kind of a pain to remember to bump 
on every new release.


At least some in the stdlib have not been.  For re.__version__, not 
since Nov 2001.



--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3UNG6KD3JIBHSMWILZGGIFHRSQV6PDKG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] In support of PEP 649

[Samuel Colvin]

I've read the recent discussions

regarding PEP 649 and PEP 563 with interest, Larry Hastings recently
contacted me when prompted

to do so in a related discussion.

I maintain pydantic  which uses type
annotations to provide runtime data validation.

I'm extremely worried that if PEP 649 gets rejected, pydantic will not be
able to fully support python 3.10 and may even end up getting abandoned, at
best it will be much slower and more brittle.

Please, please put pragmatism over principle and accept PEP 649. As
discussed in this issue
, using
`typing.get_type_hints()` is not a good replacement for type annotations
that are accessible as python objects.

I know this is not a popularity contest, but I think it's worth explaining
how widely used pydantic is for those who don't use it or haven't heard of
it:

   - it's downloaded ~7m times a month and is the most popular standalone
   python data validation library by stars
   - it's used (in production and open source projects) by most of the big
   tech companies you've heard of - it's widely used both in web development
   and data science
   - it is one of the foundations of FastAPI
 which
   is by far the fastest growing python web framework and recently came third
   (behind django and flask) in web frameworks by usage in the python
   developer survey
   

Pydantic is not the only one, there are numerous other libraries that use
type annotations at runtime.

In short, you may not like it, but runtime inspection of type annotations
makes python better, and a massive and growing number of people rely on
that.

Rejecting PEP 649 would put that in peril, please accept it.

Samuel
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7VMJWFGHVXDSRQFHMXVJKDDOVT47B54T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Simon Cross]

Replying to the original email because my comment is more a thought
after reading the whole thread rather than a response to any
particular message.

What I love about __version__ is that it is simple. It doesn't come
with the complexity of solutions like `importlib.metadata.version`.
The maintainers of the project arranged for it to be set**. It doesn't
depend on how the module was packaged or distributed.

I also don't think it's really a language update or requires any
changes in the stdlib -- it's just a standard for the community to
follow where none currently exists.

It's also probably far from the end of the world*** if it doesn't
happen, but I still like the idea.

** It might have been updated somewhat automatically -- my point here
is that distutils, setuptools, packaging, pip, wheels, PyPI were all
not involved.

*** Although that is likely true of most PEPs, even much more
complicated ones. :)

Schiavo,
Simon
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O2LGPEIOZMJYDJ4GMZXMOQK7GDWYS6X5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Barry Warsaw]

On Apr 15, 2021, at 09:54, Antoine Pitrou  wrote:
> 
> On Thu, 15 Apr 2021 09:38:41 -0700
> Barry Warsaw  wrote:
>> On Apr 14, 2021, at 23:11, Christopher Barker  wrote:
>>> 
>>> You wrote the original PEP, so of course you can withdraw it (or reject 
>>> it), but...
>>> 
>>> Are you sure? See this discussion, I don't think it's as simple as all that.
>> 
>> From a library maintainers point of view, I personally want to get away from 
>> using __version__ strings at all.  They’re kind of a pain to remember to 
>> bump on every new release.
> 
> This is often handled using `setuptools_scm` or `versioneer` (see PyPI
> for each of these packages).

Yep, exactly.  (I knew about and had in mind setuptools_scm but I didn’t know 
about versioneer; thanks for that reference!)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4SGZFUP5OD2XPRAV7PLDYGYZFISKOWIN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Antoine Pitrou]

On Thu, 15 Apr 2021 10:28:53 -0700
Brett Cannon  wrote:
> On Thu, Apr 15, 2021 at 8:50 AM Christopher Barker 
> wrote:
> 
> > On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner 
> > wrote:
> >  
> >> Paul Bryan:  
> >> > Seems like this is something that should make its way into stdlib?  
> >>
> >> In the last 10 years, the trend is more to remove anything related to
> >> packaging *outside* the stdlib :-)  
> >
> >
> > Well, maybe all of us don't think that's a good idea ;-)
> >  
> 
> As an active participant in the PyPA side of the packaging ecosystem and
> one of the maintainers of 'packaging', I am definitely NOT in favour of
> moving any of 'packaging' into the stdlib, nor breaking it up. We are
> taking distutils out of the stdlib for a reason. Backsliding on plans that
> have been in place for years is not something I would want to see happen as
> the motivations have not changed.

This seems gratuitously dogmatic. Version numbers are a simple feature
that is stably encoded in PEP 440. It's purely computational, does
not involve reading or saving any persistent state in the filesystem,
making any network access, etc. Why wouldn't it have its place in the
stdlib?

> > But anyway, I would say removing anything *related* to packaging outside
> > the stdlib is a bad idea -- requiring an external tool to build a package
> > is OK, but requireing an external use packages, not so much.
> >  
> 
> You can totally use a package already as things exist today and will in the
> future; you're just wanting a specific way to interpret a package's
> metadata which I don't view as the same thing.

That specific way is a Python standard (PEP 440). Having the
functionality in the stdlib would encourage people to use it. Not
having it in the stdlib encourages people to use adhoc version parsing,
or worse, naive string comparison.

> So asking every Python project to set a
> __version__ isn't going to change accessibility of version numbers when it
> comes to installed projects.

That doesn't have much to do with the suggestion of shipping a Version
class in the stdlib, though. Many projects already provide a
__version__, and that field is often inspected programmatically in
dependent packages (to work around behaviour changes, for example).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JAZRZL4FPJ7BW763TVKZVPY2OQBHQI5V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]

[Thomas Wouters]

Irit, Yury and Guido,

Thanks for submitting PEP 654 (Exception Groups and except *). The Steering
Council isn’t done debating this PEP, but we wanted to give an update so
everyone knows what to expect.

We are mostly in favour of the PEP, but we’re worried about the timing --
not just because there is still some discussion going on, but also the time
available for us to discuss the proposal, and to validate the new API once
it’s accepted. We are rapidly approaching  feature freeze for Python 3.10,
and while we’re not particularly worried about the implementation of this
PEP, we would like to see actual usage of the new API with enough time to
tweak it before the official release. That is to say, we would like to see
the "analysis of how exception groups will likely be used in asyncio
programs" turn into "here is how exception groups are used in asyncio
programs" before the final release, like in the asyncio module and the
standard library (for example, with TaskGroups). There’s also a
presentation scheduled in the PyCon US Language Summit, which may lead to
more discussion of the PEP and perhaps evolution of its ideas.

What would you and others feel about postponing the PEP until Python 3.11,
getting it in as soon as the branch opens, so that potential users of the
API -- asyncio or otherwise -- can provide us with practical experience to
validate some of the assumptions made in the proposal, and to discover any
warts or pitfalls we missed or misjudged? I don’t mean that as a slight on
the PEP, which is thoughtful, thorough and excellently written, but as a
practical observation of the complexity involved. Most of the PEP is more
of a building block for other APIs than something many users will be
directly exposed to, and we want to make sure the end result isn’t affected
by unanticipated effects of these design decisions. (This would potentially
also give third-party users more time to evaluate the API, although we know
adoption of alpha releases isn’t big enough to make this a very strong
argument for waiting.)

Like I said, we’re still discussing the PEP, so this hasn’t been decided.
If you or others feel strongly that the benefit of postponing is too small
(or the cost too high), we’ll take that into consideration. The SC has been
somewhat busy of late, and we don’t want to let our schedule be the
deciding factor here, but this may come down to there just not being enough
time before the feature freeze to make a thoughtful decision.

For the SC,
Thomas.
-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OQYVSR5JC53DILAPGKYUVKKDCTFRBBE5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Brett Cannon]

On Thu, Apr 15, 2021 at 8:50 AM Christopher Barker 
wrote:

> On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner 
> wrote:
>
>> Paul Bryan:
>> > Seems like this is something that should make its way into stdlib?
>>
>> In the last 10 years, the trend is more to remove anything related to
>> packaging *outside* the stdlib :-)
>
>
> Well, maybe all of us don't think that's a good idea ;-)
>

As an active participant in the PyPA side of the packaging ecosystem and
one of the maintainers of 'packaging', I am definitely NOT in favour of
moving any of 'packaging' into the stdlib, nor breaking it up. We are
taking distutils out of the stdlib for a reason. Backsliding on plans that
have been in place for years is not something I would want to see happen as
the motivations have not changed.


>
> But anyway, I would say removing anything *related* to packaging outside
> the stdlib is a bad idea -- requiring an external tool to build a package
> is OK, but requireing an external use packages, not so much.
>

You can totally use a package already as things exist today and will in the
future; you're just wanting a specific way to interpret a package's
metadata which I don't view as the same thing.


>
> Presumably that's why importlib.metadata exists in the stdlib.
>

It's there because of how tightly it binds to the import system; same goes
for importlib.resources.


>
> Version is arguably useful from the package user side. As I believe Victor
> mentioned, there are two uses for version information: display to the user
> -- for which version strings are fine, or programmatic comparison -- for
> which something like the Version object is very helpful. Do we only need to
> use version information programmatically when we are creating (or
> installing) packages? I don't think so -- I know I have code that (poorly)
> does version checking programmatically.
>

No one said that's the *only* use-case, but just because there's more than
that does not mean we should move any packaging code into the stdlib to
support a zero-install use-case.

A key point I want to make here is the version metadata is in every
project's .dist-info/METADATA file. That's what importlib.metadata knows
how to read and hence why people have been suggesting to use it. The
version information is fundamental to the wheel format and in fact you
can't have a wheel without it. So asking every Python project to set a
__version__ isn't going to change accessibility of version numbers when it
comes to installed projects.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V5WJW2HRRTSUN34ZHTVV2HVSBS4MMBSX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Brett Cannon]

On Thu, Apr 15, 2021 at 9:36 AM David Mertz  wrote:

> On Thu, Apr 15, 2021 at 4:55 PM Christopher Barker 
> wrote:
>
>> Presumably that's why importlib.metadata exists in the stdlib.
>>
>
> I was so hopeful about this, but in the end... not really.  I have not
> used this capability before.  Here are a few different situations I know of:
>
> >>> import re, statistics, pandas, vaex, bs4
> >>> for mod in [re, statistics, pandas, vaex, bs4]:
> ... try:
> ... print(mod.__name__, end=" ")
> ... print(mod.__version__)
> ... except:
> ... print("__version__ not present")
> ...
> re 2.2.1
> statistics __version__ not present
> pandas 1.2.4
> vaex {'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0',
> 'vaex-hdf5': '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0',
> 'vaex-jupyter': '0.6.0', 'vaex-ml': '0.11.1'}
> bs4 4.9.3
>
> >>> for mod in [re, statistics, pandas, vaex, bs4]:
> ... try:
> ... print(mod.__name__, version(mod.__name__))
> ... except Exception as err:
> ... print(mod.__name__, repr(err))
> ...
> re PackageNotFoundError('re')
> statistics PackageNotFoundError('statistics')
> pandas 1.2.4
> vaex 4.1.0
> bs4 PackageNotFoundError('bs4')
>
> It seems like (somehow or another) importlib.metadata is doing something
> perhaps more useful for vaex.  But it is doing distinctly worse for re,
> statistics, and bs4.
>

I don't see how importlib.metadata plays into your code sample since you
didn't import it to use it?


>
> Version is arguably useful from the package user side. As I believe Victor
>> mentioned, there are two uses for version information: display to the user
>> -- for which version strings are fine, or programmatic comparison -- for
>> which something like the Version object is very helpful. Do we only need to
>> use version information programmatically when we are creating (or
>> installing) packages? I don't think so -- I know I have code that (poorly)
>> does version checking programmatically.
>>
>
And is installing a dependency that much of a burden for that code?


>
> I think the key thing that Chris and I are pointing to is that there is a
> definite use for versions that is NOT for package maintainers/creators.
> Interactive use is definitely one such case, and eyeballing something, even
> if it is the oddball use that Vaex has, can help with figuring out an
> issue.  But in actual code, I relatively often do something like this.  Or
> rather, the below is what I would find really nice to be able to do.
>
> ver = robust_version(module)
> if ver >= (5, 2, 1):
> doit_modern_style()
> elif ver < (5, 2, 1):
> doit_old_style
> else:
> doit_unversioned_style()
>
> I'd like code like that to ALWAYS succeed.  No exceptions raised.  And it
> would *usually* use reasonable heuristics to figure out some sort of
> structured version info as well as is possible.  I.e. probably look in
> several places and return some custom object that represents a "best
> effort."  That object might be like NaN in being neither less than nor more
> than other things, in the fallback case.
>

All doable with importlib.metadata and 'packaging'.

-Brett


>
> --
> The dead increasingly dominate and strangle both the living and the
> not-yet born.  Vampiric capital and undead corporate persons abuse
> the lives and control the thoughts of homo faber. Ideas, once born,
> become abortifacients against new conceptions.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4575HZHNN45FNOTKH5DSD6NKNRUJGXFU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UTTUEIUVQHAIYJSIT3SSS4PBVZX22V7I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Antoine Pitrou]

On Thu, 15 Apr 2021 09:38:41 -0700
Barry Warsaw  wrote:
> On Apr 14, 2021, at 23:11, Christopher Barker  wrote:
> > 
> > You wrote the original PEP, so of course you can withdraw it (or reject 
> > it), but...
> > 
> > Are you sure? See this discussion, I don't think it's as simple as all 
> > that.  
> 
> From a library maintainers point of view, I personally want to get away from 
> using __version__ strings at all.  They’re kind of a pain to remember to bump 
> on every new release.

This is often handled using `setuptools_scm` or `versioneer` (see PyPI
for each of these packages).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J74GLEQS4HHSUQU3WO57GAURKQWWTTJP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Barry Warsaw]

On Apr 14, 2021, at 23:11, Christopher Barker  wrote:
> 
> You wrote the original PEP, so of course you can withdraw it (or reject it), 
> but...
> 
> Are you sure? See this discussion, I don't think it's as simple as all that.

From a library maintainers point of view, I personally want to get away from 
using __version__ strings at all.  They’re kind of a pain to remember to bump 
on every new release.

PEP 396 has been dormant for a long time and I have no interest in pushing it 
forward any more.  If someone else wants to take up the cause, I recommend 
creating a new PEP and referring back to 396.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WEO6KQ4YZOYNUJKSXEKSNFQP6PE4H5UK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[David Mertz]

On Thu, Apr 15, 2021 at 4:55 PM Christopher Barker 
wrote:

> Presumably that's why importlib.metadata exists in the stdlib.
>

I was so hopeful about this, but in the end... not really.  I have not used
this capability before.  Here are a few different situations I know of:

>>> import re, statistics, pandas, vaex, bs4
>>> for mod in [re, statistics, pandas, vaex, bs4]:
... try:
... print(mod.__name__, end=" ")
... print(mod.__version__)
... except:
... print("__version__ not present")
...
re 2.2.1
statistics __version__ not present
pandas 1.2.4
vaex {'vaex': '4.1.0', 'vaex-core': '4.1.0', 'vaex-viz': '0.5.0',
'vaex-hdf5': '0.7.0', 'vaex-server': '0.4.0', 'vaex-astro': '0.8.0',
'vaex-jupyter': '0.6.0', 'vaex-ml': '0.11.1'}
bs4 4.9.3

>>> for mod in [re, statistics, pandas, vaex, bs4]:
... try:
... print(mod.__name__, version(mod.__name__))
... except Exception as err:
... print(mod.__name__, repr(err))
...
re PackageNotFoundError('re')
statistics PackageNotFoundError('statistics')
pandas 1.2.4
vaex 4.1.0
bs4 PackageNotFoundError('bs4')

It seems like (somehow or another) importlib.metadata is doing something
perhaps more useful for vaex.  But it is doing distinctly worse for re,
statistics, and bs4.

Version is arguably useful from the package user side. As I believe Victor
> mentioned, there are two uses for version information: display to the user
> -- for which version strings are fine, or programmatic comparison -- for
> which something like the Version object is very helpful. Do we only need to
> use version information programmatically when we are creating (or
> installing) packages? I don't think so -- I know I have code that (poorly)
> does version checking programmatically.
>

I think the key thing that Chris and I are pointing to is that there is a
definite use for versions that is NOT for package maintainers/creators.
Interactive use is definitely one such case, and eyeballing something, even
if it is the oddball use that Vaex has, can help with figuring out an
issue.  But in actual code, I relatively often do something like this.  Or
rather, the below is what I would find really nice to be able to do.

ver = robust_version(module)
if ver >= (5, 2, 1):
doit_modern_style()
elif ver < (5, 2, 1):
doit_old_style
else:
doit_unversioned_style()

I'd like code like that to ALWAYS succeed.  No exceptions raised.  And it
would *usually* use reasonable heuristics to figure out some sort of
structured version info as well as is possible.  I.e. probably look in
several places and return some custom object that represents a "best
effort."  That object might be like NaN in being neither less than nor more
than other things, in the fallback case.

-- 
The dead increasingly dominate and strangle both the living and the
not-yet born.  Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4575HZHNN45FNOTKH5DSD6NKNRUJGXFU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Thu, Apr 15, 2021 at 2:40 AM Victor Stinner  wrote:

> Paul Bryan:
> > Seems like this is something that should make its way into stdlib?
>
> In the last 10 years, the trend is more to remove anything related to
> packaging *outside* the stdlib :-)


Well, maybe all of us don't think that's a good idea ;-)

But anyway, I would say removing anything *related* to packaging outside
the stdlib is a bad idea -- requiring an external tool to build a package
is OK, but requireing an external use packages, not so much.

Presumably that's why importlib.metadata exists in the stdlib.

Version is arguably useful from the package user side. As I believe Victor
mentioned, there are two uses for version information: display to the user
-- for which version strings are fine, or programmatic comparison -- for
which something like the Version object is very helpful. Do we only need to
use version information programmatically when we are creating (or
installing) packages? I don't think so -- I know I have code that (poorly)
does version checking programmatically.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LXLG7RPUOCY3J63AFDCAOUL5HHAO33XS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Boundaries between numbers and identifiers

[Damian Shaw]

This isn't a "professional" or probably even "valid" use case for Python
but one area this behavior is heavily used is code golf. For those not
familiar with code golf is a type of puzzle where the objective is to
complete a set of requirements in the least number of source code
characters as possible. Out of mainstream languages Python is surprisingly
good code golf.

This is just for fun puzzle solving and not a reason to keep or change
syntax in any particular way, in fact succeeding at code golf may even be
loosely correlated to bad syntax rules as puzzles tend to be completed in
one of the least readable ways a language can be written in. But at least
be aware if this becomes forbidden syntax that's likely the most affected
area of Python usage.

But it also made me think it could affect code minifiers, which is
apparently a real use case in Python:
https://github.com/dflook/python-minifier (Seems this minifier doesn't
actually remove the spaces between numbers and keywords where is could but
fascinating niche of Python I did not know about)

Regards
Damian
(he/him)



On Wed, Apr 14, 2021 at 7:56 AM Victor Stinner  wrote:

> Also, would it be possible to enhance to tokenizer to report a
> SyntaxWarning, rather than a SyntaxError?
>
> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CH7SLXKIKX47KVCWEJEMOB35BCIM7Y5U/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JBSUS2VIEQASY45SR4YKYTZ5PLU7HQ5X/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

[Ronald Oussoren via Python-Dev]

> On 15 Apr 2021, at 14:48, Antoine Pitrou  wrote:
> 
> On Thu, 15 Apr 2021 14:32:05 +0200
> Victor Stinner  wrote:
>> SOURCE_DATE_EPOCH is not a random variable, but is a *standardised*
>> environment variable:
>> https://reproducible-builds.org/docs/source-date-epoch/
> 
> Standardized by whom? This is not a POSIX nor Windows standard at
> least. Just because a Web page claims it is standardized doesn't mean
> that it is.
> 
>> More and more projects adopted it. As I wrote, the Python stdlib
>> already uses it in compileall and py_compile modules.
> 
> Those are higher-level modules.  Doing it in the gzip module directly
> sounds like the wrong place.

I agree. According to the documentation this variable is meant to be used for 
build tools to accomplish reproducible builds. This should IMHO not affect 
lower level APIs and libraries that aren’t build related. 

Ronald

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZVMT7A46CAR7CAS5MJNO5IZRZGOP6XN6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

[Antoine Pitrou]

On Thu, 15 Apr 2021 14:32:05 +0200
Victor Stinner  wrote:
> SOURCE_DATE_EPOCH is not a random variable, but is a *standardised*
> environment variable:
> https://reproducible-builds.org/docs/source-date-epoch/

Standardized by whom? This is not a POSIX nor Windows standard at
least. Just because a Web page claims it is standardized doesn't mean
that it is.

> More and more projects adopted it. As I wrote, the Python stdlib
> already uses it in compileall and py_compile modules.

Those are higher-level modules.  Doing it in the gzip module directly
sounds like the wrong place.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRLYIG7SMN2KJFFAOWMW6HQBIR3WQHNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

[Victor Stinner]

SOURCE_DATE_EPOCH is not a random variable, but is a *standardised*
environment variable:
https://reproducible-builds.org/docs/source-date-epoch/

This page explains the rationale. See the “Lying about the time” /
“violates language spec” section ;-)

More and more projects adopted it. As I wrote, the Python stdlib
already uses it in compileall and py_compile modules.

Victor

On Thu, Apr 15, 2021 at 12:34 PM Antoine Pitrou  wrote:
>
> On Thu, 15 Apr 2021 11:28:03 +0200
> Victor Stinner  wrote:
> > If gzip is modified to use SOURCE_DATE_EPOCH timestamp, you get a
> > reproducible binary and you don't need to add a new constant ;-)
> > SOURCE_DATE_EPOCH is a timestamp: number of seconds since Unix Epoch
> > (January 1, 1970 at 00:00).
>
> Changing the behaviour of a stdlib module based on an environment
> variable sounds a bit undesirable.  That behaviour can be implemented
> at a higher-level in application code (for example the tarfile or
> zipfile command line).
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/HPX62SVAMT6ELIKCDWE2JDTY4ATX2NKU/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MHDIQZZXQJRBSXDMQKV4JYR6J5UU2OFH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

[Antoine Pitrou]

On Thu, 15 Apr 2021 11:28:03 +0200
Victor Stinner  wrote:
> If gzip is modified to use SOURCE_DATE_EPOCH timestamp, you get a
> reproducible binary and you don't need to add a new constant ;-)
> SOURCE_DATE_EPOCH is a timestamp: number of seconds since Unix Epoch
> (January 1, 1970 at 00:00).

Changing the behaviour of a stdlib module based on an environment
variable sounds a bit undesirable.  That behaviour can be implemented
at a higher-level in application code (for example the tarfile or
zipfile command line).

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HPX62SVAMT6ELIKCDWE2JDTY4ATX2NKU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Antoine Pitrou]

On Thu, 15 Apr 2021 11:37:28 +0200
Victor Stinner  wrote:
> Paul Bryan:
> > Seems like this is something that should make its way into stdlib?  
> 
> In the last 10 years, the trend is more to remove anything related to
> packaging *outside* the stdlib :-) Since it's evolving way faster

What is "evolving way faster"?  packaging.version implements PEP 440,
which has been approved in 2014.  Recent changes have been really minor:
https://github.com/pypa/packaging/commits/main/packaging/version.py



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NFPGSSDTD7SNZI4XJF542NS4R7OXVVMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Victor Stinner]

Paul Bryan:
> Seems like this is something that should make its way into stdlib?

In the last 10 years, the trend is more to remove anything related to
packaging *outside* the stdlib :-) Since it's evolving way faster than
Python releases and the stdlib cannot be updated once installed, I
don't think that it's a good idea.

On Thu, Apr 15, 2021 at 12:22 AM Antoine Pitrou  wrote:
> Tangentially, until now projects could use distutils's LooseVersion if
> they wanted to compare version numbers reliably.  With distutils being
> deprecated, they'll have to either depending on packaging (which is a
> large dependency just for comparison version numbers) or vendor
> packaging's Version class (which is doable but still some bothersome
> additional work).

If packaging is too big and if packaging maintainters like the idea,
maybe packaging.version could be moved into a dedicated package? I
didn't look if it makes sense from a technical point of view.

$ wc -l packaging/version.py  packaging/_structures.py
  556 packaging/version.py
   86 packaging/_structures.py
  642 total

version.py uses _structures.py (InfinityType, NegativeInfinityType).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GT6NZDIKA222ADKKSJN7W4WZROMAIRDF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: gzip.py: allow deterministic compression (without time stamp)

[Victor Stinner]

If gzip is modified to use SOURCE_DATE_EPOCH timestamp, you get a
reproducible binary and you don't need to add a new constant ;-)
SOURCE_DATE_EPOCH is a timestamp: number of seconds since Unix Epoch
(January 1, 1970 at 00:00).

Victor


On Wed, Apr 14, 2021 at 8:15 PM  wrote:
>
> The gzip specification [1] makes clear that the mtime field is always present.
> The time is in Unix format, i.e., seconds since 00:00:00 GMT, Jan.  1, 1970.
> MTIME = 0 means no time stamp is available. Hence no need for a
> new constant NO_TIMESTAMP.
>
> So this is primarily a documentation problem [2]. For this, I will create a
> pull request to gzip.py.
>
> Joachim
>
> [1] https://www.ietf.org/rfc/rfc1952.txt
> [2] 
> https://discuss.python.org/t/gzip-py-allow-deterministic-compression-without-time-stamp
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/LCPWERWIFG4AJS6DPHNEMOGBYJ2APDJ3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FZTBVELC53IX6CGRCG53IGECJC3SANLE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Inada Naoki]

I updated the benchmark little:

* Added no annotation mode for baseline performance.
* Better stats output.

https://gist.github.com/methane/abb509e5f781cc4a103cc450e1e7925d

```
# No annotation (master + GH-25419)
$ ./python ~/ann_test.py 0
code size: 102967 bytes
memory: 181288 bytes
unmarshal: avg: 299.301ms +/-1.257ms
exec: avg: 104.019ms +/-0.038ms

# PEP 563 (master + GH-25419)
$ ./python ~/ann_test.py 2
code size: 110488 bytes
memory: 193572 bytes
unmarshal: avg: 313.032ms +/-0.068ms
exec: avg: 108.456ms +/-0.048ms

# PEP 649 (co_annotations + GH-25419 + GH-23056)
$ ./python ~/ann_test.py 3
code size: 204963 bytes
memory: 209257 bytes
unmarshal: avg: 587.336ms +/-2.073ms
exec: avg: 97.056ms +/-0.046ms

# Python 3.9
$ python3 ann_test.py 0
code size: 108955 bytes
memory: 173296 bytes
unmarshal: avg: 333.527ms +/-1.750ms
exec: avg: 90.810ms +/-0.347ms

$ python3 ann_test.py 1
code size: 121011 bytes
memory: 385200 bytes
unmarshal: avg: 334.929ms +/-0.055ms
exec: avg: 400.260ms +/-0.249ms
```

## Rough estimation of annotation overhead

Python 3.9 w/o PEP 563
code (pyc) size: +11%
memory usage: +122%  (211bytes / function)
import time: +73% (*)

PEP 563
code (pyc) size: +7.3%
memory usage: +0.68%  (13.3bytes / function)
import time: +4.5%

PEP 649
code (pyc) size: +99%
memory usage: +15%  (28 bytes / function)
import time: +70%

(*) import time can be much more slower for complex annotations.

## Conclusion

* PEP 563 is close to "zero overhead" in memory consumption. And
import time overhead is ~5%. Users can write type annotations without
caring overhead.

* PEP 649 is much better than old semantics for memory usage and
import time. But import time is still longer than no annotation code.

  * The import time overhead is coming from unmarshal, not from
eval().  If we implement a "lazy load" mechanizm for docstrings and
annotations, overhead will become cheaper.
  * pyc file become bigger (but who cares?)

I will read PEP 649 implementation to find missing optimizations other
than GH-25419 and GH-23056.

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5PFGS6DQUHUG63UWRXYNLLMXAVELP32/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Wed, Apr 14, 2021 at 2:23 PM Paul Moore  wrote:

> >> because PEP 440 versions are fundamental to packaging.
> >
> > Are you suggesting that users should have to install an external module
> to tell what version of packages they are using?!
>
> No. To tell what version of a package they are using, a string is
> sufficient.
>

exactly -- if we do anything at all here, it should probably be at least.

if you provide a __version__ attribute, it should be a PEP 440 compliant
string.

That would at least make the use __version__ a touch more consistent.

The next, more controversial, step would be to suggest that people SHOULD
provide a __version__string on all top-level packages. I would like that --
the current situation where many, but not all packages have __version__ is
really annoying.

They only need a version object if they want to do additional
> processing (like comparing versions, or checking whether a version
> meets a constraint).
>

And indeed, we could add the ability for packaging.version.Version objects
to be able to compare themselves with compatible strings -- I think that
would be pretty handy.

Given that the packaging ecosystem already has a canonical version
> object (provided by the packaging library), which has been used and
> tested extensively in many environments, inventing a different API
> seems at best ill-advised.


absolutely.


> Whether the stdlib needs a version object.
> rather than leaving that functionality to a 3rd party library, is the
> same question that comes up for *any* functionality that's proposed
> for the stdlib, and I have no particular opinion in this case.
>

I don't think it's the same as any functionality -- if we do want to better
standardize package versioning in Python, and I think we should, then the
Version object may become something useful to, and maybe even needed by,
virtually every third party package. Which makes it a pretty prime
candidate for the stdlib.

Alternatively,  the packaging package is pretty small, but if it grows, it
might be good to split out the run-time vs build-time pieces.

It's designed for programmatic use, not interactive use, yes. But
> that's sort of my point, why do you want anything more than the bare
> string in the REPL? What are you planning on doing with it?
>

there is something in between the REPL and full on system development --
something simple for quickly scripts is nice too. But a simple standardised
version string is fine for that.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BWIF72TLNB6A63SXYUO3KMSR6ZUNTALS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Revive PEP 396 -- Module Version Numbers ?

[Christopher Barker]

On Wed, Apr 14, 2021 at 12:10 PM Barry Warsaw  wrote:

I’d forgotten that this PEP was in Deferred state.  I think it should be
> rejected and I plan on making that change.  importlib.metadata is a much
> better approach to programmatically determining package versions.
>
>
> https://docs.python.org/3/library/importlib.metadata.html#distribution-versions


Barry,

You wrote the original PEP, so of course you can withdraw it (or reject
it), but...

Are you sure? See this discussion, I don't think it's as simple as all that.

Honestly, I'm still kind of confused about a "distribution" vs a package or
module, but in general, it seems to me that the packaging community has
done a great job of building a system that can accommodate a lot of
complexity, but as they say:

The easy things should be easy, and the hard things should be possible.

And the full process of using packaging and distributions, and a Version
object and all that is not so easy anymore --at least compared to putting:

__version__ = "1.2.3"

in a package __init__.py.

Anyway, a couple points:

1) it seems self-evident that it would be easier for the whole community if
"how to declare", and "how to find" the version of a package was
standardized: "there's only one way to do it"

1b) It seems like a really good idea that the (or one of the) official
recommended way to do it is supported in the standard library itself, and
using a string is pretty easy to support :-) -- do we really want people to
have to install a third party package to set or get the version of a simple
module?

2) __version__ strings are already pretty darn common -- maybe more common
than proper distribution metadata? -- are we saying that everyone should
stop doing that?

3) I don't know that having __version__ strings is incompatible with
packging.version.Version or importlib.metadata.version

4) Yes, there are real advantages to the more complex system for
programmatically working with versions. But there are real advantages to
something simple for human readability / interactive use -- and, frankly,
simple scripting. See David Mertz's recent post -- I went through a similar
process in iPython.

Consider this analogy: if I want to know what version of git I have
installed, I can run:

$ git --version

And that's a convention adhered to by many *nix command line tools. Would
we all really be happier if we had to go use an rpm or deb or homebrew, or
?? command to see what package i have installed? I think this is similar --
while a good distribution management system is a good thing, it's also good
if I can tell something about the version of a package/module from
the package itself.

With regard to having both __version__ and importlib.metadata.version, see
beautiful soup for example:

In [10]: import beautifulsoup4
...
ModuleNotFoundError: No module named 'beautifulsoup4'

oops, that's not what the importable module is called. So:

In [11]: import bs4

That worked -- now, what version is it?

In [12]: importlib.metadata.version('bs4')
---
PackageNotFoundError  Traceback (most recent call last)
...
PackageNotFoundError: bs4

[*]

That didn't work -- I guess I need to use the distribution name:

In [13]: importlib.metadata.version('beautifulsoup4')
Out[13]: '4.9.3'

OK, that worked.

But I imported "bs4", so what if I do this?

In [14]: bs4.__version__
Out[14]: '4.9.3'

Ah yes, that worked too.

So I guess what I'm suggesting is that we officially recommend that folks
do what the BeautifulSoup folks, and many others, are already doing. In
many (most) cases, the distribution name and the importable package name
are the same, but this does accommodate both.

[*] One other annoyance to importlib.metadata.version is that in addition
to having to import importlib to use it, I also need to use
importlib.metadata.PackageNotFoundError if I want to trap the exception.

BTW: I'm not making that last point up -- I recently updated some in-house
code (that someone else had written) in a web app that presented the
version of various packages core to the system. We had just refactor and
one of these packages what now optional, but the whole system was crashing
on that darn PackageNotFoundError. I ended up jsut commenting out that
package, rather than dealing with how to catch the exception and deal with
it properly. I would have so much rathered a simple:

try:
import the_package
the_pacakge_version = the_package.__version__
except
the_package_version = None

Granted, it's not that hard to "do it right", but I still like things
simple :-)

Anyway, as the original PEP author is no longer supportive, this is dead in
the water unless another core dev is interested in sponsoring this (or a
new) PEP. If not, I can stop wasting my and everyone else's time.

-Chris

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop