[Python-Dev] Re: Proto-PEP part 4: The wonderful third option
Hi Paul, On Sun, May 1, 2022 at 3:47 PM Paul Bryan wrote: > > Can someone state what's currently unpalatable about 649? It seemed to > address the forward-referencing issues, certainly all of the cases I was > expecting to encounter. Broadly speaking I think there are 3-4 issues to resolve as part of moving forward with PEP 649: 1) Class decorators (the most relevant being @dataclass) that need to inspect something about annotations, and because they run right after class definition, the laziness of PEP 649 is not sufficient to allow forward references to work. Roughly in a similar boat are `if TYPE_CHECKING` use cases where annotations reference names that aren't ever imported. 2) "Documentation" use cases (e.g. built-in "help()") that really prefer access to the original text of the annotation, not the repr() of the fully-evaluated object -- this is especially relevant if the annotation text is a nice short meaningful type alias name, and the actual value is some massive unreadable Union type. 3) Ensuring that we don't regress import performance too much. 4) A solid migration path from the status quo (where many people have already started adopting PEP 563) to the best future end state. Particularly for libraries that want to support the full range of supported Python versions. Issues (1) and (2) can be resolved under PEP 649 by providing a way to run the __co_annotations__ function without erroring on not-yet-defined names, I think we have agreement on a plan there. Performance of the latest PEP 649 reference implementation does not look too bad relative to PEP 563 in my experiments, so I think this is not an issue -- there are ideas for how we could reduce the overhead even further. The migration path is maybe the most difficult issue -- specifically how to weigh "medium-term migration pain" (which under some proposals might last for years) vs "best long-term end state." Still working on reaching consensus there, but we have options to choose from. Expect a more thorough proposal (probably in the form of an update to PEP 649?) sometime after PyCon. Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M3FB6QHB2IOMEXDGHFRHYQEDR3KGZPHG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Tue, Apr 26, 2022 at 7:24 PM Greg Ewing wrote: > On 27/04/22 2:01 am, Chris Angelico wrote: > > That would be the case if monkeypatching were illegal. Since it's not, > > wherein lies the difference? > > The proposed feature is analogous to forward declaring a > struct in C. Would you call what C does monkeypatching? > It is not analogous; it is a false analogy that obscures the issues with this proposal in Python. A C forward declaration (not to mention the full struct declaration also!) is purely for the compiler; at runtime one can have a pointer to some memory that the compiler expects to be shaped like that struct, but one can never get hold of any runtime value that is “the struct definition itself,” let alone a runtime value that is the nascent forward-declared yet-to-be-completed struct. So clearly there can be no patching of something that never exists at runtime at all. Python is quite different from C in this respect. Classes are runtime objects, and so is the “forward declared class” object. The proposal is for a class object to initially at runtime be the latter, and then later (at some point that is not well defined if the implementation is in a separate module, because global module import ordering is an unstable emergent property of all the imports in the entire codebase) may suddenly, everywhere and all at once, turn into the former. Any given module that imports the forward declared name can have no guarantee when (if ever) that object will magically transform into something that is safe to use. Whether we call it monkeypatching or not is irrelevant. Having global singleton objects change from one thing to a very different thing, at an unpredictable point in time, as a side effect of someone somewhere importing some other module, causes very specific problems in being able to locally reason about code. I think it is more useful to discuss the specific behavior and its consequences than what it is called. Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CQ7TAV6TWGEG2HLVY7T46U6JCPESRACR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 4: The wonderful third option
On Tue, Apr 26, 2022 at 1:25 PM Guido van Rossum wrote:
> I also would like to hear more about the problem this is trying to solve,
> when th real-world examples. (E.g. from pydantic?)
Yes please. I think these threads have jumped far too quickly into
esoteric details of implementation and syntax, without critical
analysis of whether the semantics of the proposal are in fact a good
solution to a real-world problem that someone has.
I've already outlined in a more detailed reply on the first thread why
I don't think forward declarations provide a practically useful
solution to forward reference problems for users of static typing
(every module that imports something that might be a forward reference
would have to import its implementation also, turning every one-line
import of that class into two or three lines) and causes new problems
for every user of Python due to its reliance on import side effects
causing global changes at a distance. See
https://mail.python.org/archives/list/python-dev@python.org/message/NMCS77YFM2V54PUB66AXEFTE4NXFHWPI/
for details.
Under PEP 649, forward references are a small problem confined to the
edge case of early resolution of type annotations. There are simple
and practical appropriately-scoped solutions easily available for that
small problem: providing a way to resolve type annotations at runtime
without raising NameError on not-yet-defined names. Such a facility
(whether default or opt-in) is practically useful for many users of
annotations (including dataclasses and documentation tools), which
have a need to introspect some aspects of annotations without
necessarily needing every part of the annotation to resolve. The
existence of such a facility is a reasonable special case for
annotations specifically, because annotations are fundamentally
special: they provide a description of code, rather than being only a
part of the code. (This special-ness is precisely also why they cause
more forward references in the first place.)
IMO, this forward declaration proposal takes a small problem in a
small corner of the language and turns it into a big problem for the
whole language, without even providing as nice and usable an option
for common use cases as "PEP 649 with option for lax resolution" does.
This seems like a case study in theoretical purity ("resolution of
names in annotations must not be special") running roughshod over
practicality.
Carl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/RVQSLD435BFKEVIMY2AIA5MCJB37BPHK/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Sun, Apr 24, 2022 at 10:20 AM Joao S. O. Bueno wrote: > > I am not worried about the bikeshed part of which syntax to use - > and more worried with the possible breaking of a lot of stuff, unless > we work with creation of a non-identical "forward object" that is > rebound, as in plain name binding, when the second part > is declared. I've stated that amidst my ramblings, > but Nick Coghlan managed to keep it suscint at > https://mail.python.org/archives/list/python-dev@python.org/message/DMITVTUIQKJW6RYVOPQXHD54VSYE7QHA/ I don't think name rebinding works. That means that if we have `forward class Bar` in module `foo` and `continue class Bar: ...` in module `foo.impl`, if module `baz` does `from foo import Bar`, it will forever have either the forward reference object or the real class, and which one it has is entirely unpredictable (depends on import ordering accidents of the rest of the codebase.) If `baz` happens to be imported before `foo.impl`, the name `Bar` in the `baz` namespace will never be resolved to the real class, and isn't resolvable to the real class without some outside intervention. > """ > Something worth considering: whether forward references need to be > *transparent* at runtime. If they can just be regular ForwardRef objects > then much of the runtime complexity will go away (at the cost of some > potentially confusing identity check failures between forward references > and the actual class objects). > > ForwardRef's constructor could also potentially be enhanced to accept a > "resolve" callable, and a "resolve()" method added to its public API, > although the extra complexity that would bring may not be worth it. > """ I'm not sure how this `resolve()` method is even possible under the proposed syntax. If `forward class Bar` and `continue class Bar` are in different modules, then how can `forward class Bar` (which must create the "forward reference" object) possibly know which module `continue class Bar: ...` exists in? How can it know how to resolve itself? Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WLZRZIMPRST52UMINB5VB57TOIQVTYQH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
Hi Larry, On Sat, Apr 23, 2022 at 1:53 AM Larry Hastings wrote: > But rather than speculate further, perhaps someone who works on one of the > static type analysis checkers will join the discussion and render an informed > opinion about how easy or hard it would be to support "forward class" and > "continue class". I work on a Python static type checker. I think a major issue with this proposal is that (in the separate-modules case) it requires monkey-patching as an import side effect, which is quite hard for both humans and static analysis tools to reason effectively about. Imagine we have a module `foo` that contains `forward class Bar`, a separate module `foo.impl` that contains `continue class Bar: ...`, and then a module `baz` that contains `import foo`. What type of object is `foo.Bar` during the import of `baz`? Will it work for the module body of `baz` to create a singleton instance `my_bar = foo.Bar()`? The answer is that we have no idea. `foo.Bar` might be a non-instantiable "forward class declaration" (or proxy object, in your second variation), or it might be a fully-constituted class. Which one it is depends on accidents of import order anywhere else in the codebase. If any other module happens to have imported `foo.impl` before `baz` is imported, then `foo.Bar` will be the full class. If nothing else has imported `foo.impl`, then it will be a non-instantiable declaration/proxy. This question of import order potentially involves any other module in the codebase, and the only way to reliably answer it is to run the entire program; neither a static type checker nor a reader of the code can reliably answer it in the general case. It will be very easy to write a module `baz` that does `import foo; my_bar = foo.Bar()` and have it semi-accidentally work initially, then later break mysteriously due to a change in imports in a seemingly unrelated part of the codebase, which causes `baz` to now be imported before `foo.impl` is imported, instead of after. There is another big problem for static type checkers with this hypothetical module `baz` that only imports `foo`. The type checker cannot know the shape of the full class `Bar` unless it sees the right `continue Bar: ...` statement. When analyzing `baz`, it can't just go wandering the filesystem aimlessly in hopes of encountering some module with `continue Bar: ...` in it, and hope that's the right one. (Even worse considering it might be `continue snodgrass: ...` or anything else instead.) So this means a type checker must require that any module that imports `Bar` MUST itself import `foo.impl` so the type checker has a chance of understanding what `Bar` actually is. This highlights an important difference between this proposal and languages with real forward declarations. In, say, C++, a forward declaration of a function or class contains the full interface of the function or class, i.e. everything a type checker (or human reader) would need to know in order to know how it can use the function or class. In this proposal, that is not true; lots of critical information about the _interface_ of the class (what methods and attributes does it have, what are the signatures of its methods?) are not knowable without also seeing the "implementation." This proposal does not actually forward declare a class interface; all it declares is the existence of the class (and its inheritance hierarchy.) That's not sufficient information for a type checker or a human reader to make use of the class. Taken together, this means that every single `import foo` in the codebase would have to be accompanied by an `import foo.impl` right next to it. In some cases (if `foo.Bar` is not used in module-level code and we are working around a cycle) it might be safe for the `import foo.impl` to be within an `if TYPE_CHECKING:` block; otherwise it would need to be a real runtime import. But it must always be there. So every single `import foo` in the codebase must now become two or three lines rather than one. There are of course other well-known problems with import-time side effects. All the imports of `foo.impl` in the codebase would exist only for their side effect of "completing" Bar, not because anyone actually uses a name defined in `foo.impl`. Linters would flag these imports as unused, requiring extra cruft to silence the linter. Even worse, these imports would tend to appear unused to human readers, who might remove them and be confused why that breaks the program. All of these import side-effect problems can be resolved by dis-allowing module separation and requiring `forward class` and `continue class` to appear in the same module. But then the proposal no longer helps with resolving inter-module cycles, only intra-module ones. Because of these issues (and others that have been mentioned), I don't think this proposal is a good solution to forward references. I think PEP 649, with some tricks that I've mentioned elsewhere to allow introspecting annotations
[Python-Dev] Re: Declarative imports
Hi Malthe, On Fri, Apr 8, 2022 at 12:04 PM Malthe wrote: > Actually, to me the interesting idea is not so much lazy imports – I > think they should not be lazy, at least that was my initial thought. I > think they should be immediately resolved before anything else in that > module: I'm +0.25 on your idea as simply a streamlined syntax for inline imports (given actually finding an appropriate syntax, which I haven't thought much about; @ probably doesn't work due to the conflict with decorator syntax, but there might be other options.). If it existed I would probably use it occasionally, but I don't feel a strong need for it. But I think your proposal is much stronger if you eliminate the hoisting from it; with the hoisting I'd be -1. Out-of-source-order execution like this is just quite surprising in the context of Python. > 1. This would settle any discussion about performance impact (there > wouldn't be any). If the inline import is actually a performance problem because a certain code path is very hot, the solution is simple: don't use the inline import there, use a top-of-module import instead. > 2. This would enable IDEs, typers and other tooling to know the type > using existing import logic. I don't think it enables any such thing. Static-analysis tooling has only the source code to work with, runtime behavior doesn't affect it. If the runtime executes these imports out-of-order, that won't make the slightest difference to how easily IDEs and type checkers can analyze the source code. > 3. Catch errors early! The very strong precedent in Python is that errors in code are caught when the code runs, and the code runs more or less when you'd expect it to, in source order. If you want to catch errors earlier, use a static analysis tool to help catch them. Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ARI44O62CRMAF2IKPHJVLU5D2ADR2DP6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
Hi Barry, On Fri, Apr 8, 2022 at 12:44 PM Barry Warsaw wrote: > > Start up overhead due to imports is a real problem for some class of > applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get > Python CLIs to be more responsive. E.g. getting from invocation to —help > output is a major UX problem. Definitely, we have this same problem, and also the same symptom of people pushing hard to rewrite Python CLIs in Go for this reason. > It’s often more complicated than just imports alone though. Expensive module > scope initializations and decorators contribute to this problem. One of our projects that can prevent much of this expensive work being done at import time is Strict Modules[1]. Currently it's only available as part of Cinder, though we're hoping to make it pip-installable as part of our project to make Cinder's features more easily accessible. Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.) Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2] [1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/62OTFJMAMQ2WHZ4H3TUEJTECMPJDQ557/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
An interesting point in the lazy imports design space that I hadn't previously considered could be: - lazy imports are explicitly marked and usage of the imported name within the module is transparent, but - lazily imported names are _not_ visible in the module namespace; they can't be accessed by other modules or re-exported; they are internal-use-only within the module This compromise would, I think, make it possible to implement lazy imports entirely in the compiler (effectively as syntax sugar for an inline import at every usage site), which is definitely an implementation improvement. I think in practice explicitly marking lazy imports would make it somewhat harder to gain the benefits of lazy imports for e.g. speeding up startup time in a large CLI, compared to an implicit/automatic approach. But still could be usable to get significant benefits. Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZT6CXQPFCWZD2M65YXCSAPPGNDGA6WNE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
You only get the ease-of-implementation benefit if you are willing to explicitly mark every _use_ of a lazy-imported name as special (and give the fully qualified name at every usage site). This is rather more cumbersome (assuming multiple uses in a module) than just explicitly marking an import as lazy in one location and then using the imported name in multiple places normally. Other "lazy import" solutions are trying to solve a problem where you want the name to be usable (without special syntax or marking) in many different places in a module, and visible in the module namespace always -- but not actually imported until someone accesses/uses it. The difficulty arises because in this case you need some kind of placeholder for the "deferred import", but you need to avoid this "deferred object" escaping and becoming visible to Python code without being resolved first. Explicitly marking which imports are lazy is fine if you want it (it's just a matter of syntax), but it doesn't do anything to solve the problem of allowing usage of the lazy-imported name to be transparent. I agree that the idea that top-of-module imports help readability is overstated; it sounds slightly Stockholm-syndrome-ish to me :) Top-of-module imports are frankly a pain to maintain and a pain to read (because they are often distant from the uses of the names). But they are a necessary evil if you want a) namespaces and b) not constantly retyping fully-qualified names at every usage site. Python is pretty committed to namespaces at this point (and I wouldn't want to change that), so that leaves the choice between top-of-module imports vs fully qualifying every use of every name; pick your poison. (Inline imports in a scope with multiple uses are a middle ground.) Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N4T2YMPHBLJXKCFA5CIPBFIZJJKO7SHR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
Hi Larry, On 4/14/21, 1:56 PM, "Larry Hastings" wrote: >My plan was to post it here and see what the response was first. Back in > January, when I posted the first draft, I got some very useful feedback that > resulted in some dramatic changes. This time around, so far, nobody has > suggested even minor changes. Folks have just expressed their opinions about > it (which is fine). This is not true. I suggested yesterday (in https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/ ) that PEP 649 could avoid making life worse for users of type annotations (relative to PEP 563) if it replaced runtime-undefined names with forward reference markers, as implemented in https://github.com/larryhastings/co_annotations/pull/3 Perhaps you've chosen to ignore the suggestion, but that's not the same as nobody suggesting any changes ;) Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4LXOED3ABKDSNUDJ3JTNEGTXD3R7TEWT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2
Hi Larry, On 4/12/21, 6:57 PM, "Larry Hastings" wrote: Again, by "works on PEP 563 semantics", you mean "doesn't raise an error". But the code has an error. It's just that it has been hidden by PEP 563 semantics. I don't agree that changing Python to automatically hide errors is an improvement. As the Zen says: "Errors should never pass silently." This is really the heart of the debate over PEP 649 vs PEP 563. If you examine an annotation, and it references an undefined symbol, should that throw an error? There is definitely a contingent of people who say "no, that's inconvenient for us". I think it should raise an error. Again from the Zen: "Special cases aren't special enough to break the rules." Annotations are expressions, and if evaluating an expression fails because of an undefined name, it should raise a NameError. Normally in Python, if you reference a symbol in a function definition line, the symbol must be defined at that point in module execution. Forward references are not permitted, and will raise `NameError`. And yet you have implemented PEP 649, whose entire raison d'être is to implement a "special case" to "break the rules" by delaying evaluation of annotations such that a type annotation, unlike any other expression in the function definition line, may include forward reference names which will not be defined until later in the module. The use case for `if TYPE_CHECKING` imports is effectively the same. They are just forward references to names in other modules which can't be imported eagerly, because e.g. it would cause a cycle. Those who have used type annotations in earnest are likely to confirm that such inter-module forward references are just as necessary as intra-module forward references for the usability of type annotations. So it doesn't seem that we have here is a firm stand on principle of the Zen, it appears to rather be a disagreement about exactly where to draw the line on the "special case" that we all already seem to agree is needed. The Zen argument seems to be a bit of a circular one: I have defined PEP 649 semantics in precisely this way, therefore code that works with PEP 649 does not have an error, and code that does not work with PEP 649 "has an error" which must be surfaced! With PEP 563, although `get_type_hints()` cannot natively resolve inter-module forward references and raises `NameError`, it is possible to work around this by supplying a globals dict to `get_type_hints()` that has been augmented with those forward-referenced names. Under the current version of PEP 649, it becomes impossible to get access to such type annotations at runtime at all, without reverting to manually stringifying the annotation and then using something like `get_type_hints()`. So for users of type annotations who need `if TYPE_CHECKING` (which I think is most users of type annotations), the best-case overall effect of PEP 649 will be that a) some type annotations have to go back to being ugly strings in the source, and b) if type annotation values are needed at runtime, `get_type_hints()` will still be as necessary as it ever was. It is possible for PEP 649 to draw the line differently and support both intra-module and inter-module forward references in annotations, by doing something like https://github.com/larryhastings/co_annotations/pull/3 and replacing unknown names with forward-reference markers, so the annotation values are still accessible at runtime. This meets the real needs of users of type annotations better, and gives up none of the benefits of PEP 649. Carl ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DSZFE7XTRK2ESRJDPQPZIDP2I67E76WH/ Code of Conduct: http://python.org/psf/codeofconduct/
