[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent
2022-04-09 04:24 UTC, Terry Reedy は書いた: > Perhaps something intentionally vague like > > "Manual deletion of entries from sys.modules may invalidate statements > above, even after re-imports." > > or > > "Manual deletion of entries from sys.modules may result in surprising > behavior, even after re-imports." Not only deletion, but also random assignments: >>> import sys >>> import collections.abc >>> sys.modules['collections'] = 1 >>> import collections.abc >>> collections.abc Traceback (most recent call last): File "", line 1, in AttributeError: 'int' object has no attribute 'abc' ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/IRXLI6XANNQTOGSBQGOFX25UJD6J4SGJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
Hi Barry, On Fri, Apr 8, 2022 at 12:44 PM Barry Warsaw wrote: > > Start up overhead due to imports is a real problem for some class of > applications, e.g. CLIs, and I’ve seen a lot of hacks implemented to get > Python CLIs to be more responsive. E.g. getting from invocation to —help > output is a major UX problem. Definitely, we have this same problem, and also the same symptom of people pushing hard to rewrite Python CLIs in Go for this reason. > It’s often more complicated than just imports alone though. Expensive module > scope initializations and decorators contribute to this problem. One of our projects that can prevent much of this expensive work being done at import time is Strict Modules[1]. Currently it's only available as part of Cinder, though we're hoping to make it pip-installable as part of our project to make Cinder's features more easily accessible. Our experience in practice, though, has been that universally lazy imports is somewhat easier to adopt than Strict Modules, and has had a much bigger overall impact on reducing startup time for big CLIs (and a big web server too; as you note it's not as serious an issue for a web server in production, but restart time still does make a difference to dev speed / experience.) Removing slow stuff happening at import time helps, but it'll never match the speed of not doing the import at all! We've seen startup time improvements up to 70% in real-world CLIs just by making imports lazy. We've also opened an issue to discuss the possibility of upstreaming this. [2] [1] https://github.com/facebookincubator/cinder/#strict-modules [2] https://bugs.python.org/issue46963 ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/62OTFJMAMQ2WHZ4H3TUEJTECMPJDQ557/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: About PEPs being discussed on Discourse
Brett Cannon writes: > On Thu, Apr 7, 2022 at 7:33 PM Stephen J. Turnbull < > [email protected]> wrote: > > The specific mention of "community moderation" and "thread management" > > makes me suspect that part of that effect is due to increased cost of > > participation for casual observers. > > As one of the discuss.python.org admins and a former ML admin (sans > python-committers), I can tell that "increased cost of > participation for casual observers" is not at all the motivation > behind Greg's comment I worded that post very carefully, and deny any statement about channel admins' motivation was intended or present in it. My point is entirely that the Law of Unintended Effects is valid, and may be operating here. Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/G4Y2OZCSRXOC3BAKZZ3IYPODVWSJ5CXD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: About PEPs being discussed on Discourse
Hi, Le 08/04/2022 à 15:34, Petr Viktorin a écrit : > > I follow Discourse by e-mail, in the "Mailing list mode", where > Discourse sends me all comments and I filter in the e-mail client. That > works rather well for me. If this is so, is there a possibility to funnel the message stream through GMANE (or some similar list archive), for the benefit of casual readers? (And no, the Discourse web interface without logging in just doesn't cut it for casual reading. You can't read a whole thread without being interrupted by nag screens "You seem to be interested etc…". And you can't remember what you already read and what not.) Cheers, Baptiste ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FIS4GHPEBM3M6HYCN72I63JL73AKYMNN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
Hi Malthe, On Fri, Apr 8, 2022 at 12:04 PM Malthe wrote: > Actually, to me the interesting idea is not so much lazy imports – I > think they should not be lazy, at least that was my initial thought. I > think they should be immediately resolved before anything else in that > module: I'm +0.25 on your idea as simply a streamlined syntax for inline imports (given actually finding an appropriate syntax, which I haven't thought much about; @ probably doesn't work due to the conflict with decorator syntax, but there might be other options.). If it existed I would probably use it occasionally, but I don't feel a strong need for it. But I think your proposal is much stronger if you eliminate the hoisting from it; with the hoisting I'd be -1. Out-of-source-order execution like this is just quite surprising in the context of Python. > 1. This would settle any discussion about performance impact (there > wouldn't be any). If the inline import is actually a performance problem because a certain code path is very hot, the solution is simple: don't use the inline import there, use a top-of-module import instead. > 2. This would enable IDEs, typers and other tooling to know the type > using existing import logic. I don't think it enables any such thing. Static-analysis tooling has only the source code to work with, runtime behavior doesn't affect it. If the runtime executes these imports out-of-order, that won't make the slightest difference to how easily IDEs and type checkers can analyze the source code. > 3. Catch errors early! The very strong precedent in Python is that errors in code are caught when the code runs, and the code runs more or less when you'd expect it to, in source order. If you want to catch errors earlier, use a static analysis tool to help catch them. Carl ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/ARI44O62CRMAF2IKPHJVLU5D2ADR2DP6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent
> Not only deletion, but also random assignments: > > >>> import sys > >>> import collections.abc > >>> sys.modules['collections'] = 1 > >>> import collections.abc > >>> collections.abc > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'int' object has no attribute 'abc' Sure, but I hope people expect that kind of monkey patching to break things. -CHB > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/IRXLI6XANNQTOGSBQGOFX25UJD6J4SGJ/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/MNW7VHF4DHSAMIULCTN67LOS6BEMOZXK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent
On 4/9/2022 5:09 AM, Arfrever Frehtes Taifersar Arahesis wrote: 2022-04-09 04:24 UTC, Terry Reedy は書いた: Perhaps something intentionally vague like "Manual deletion of entries from sys.modules may invalidate statements above, even after re-imports." or "Manual deletion of entries from sys.modules may result in surprising behavior, even after re-imports." Not only deletion, but also random assignments: Ok. Change "Manual deletion of entries from sys.modules" to "Direct manipulation of sys.modules" Terry Jan Reedy ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BKGWDKLV3WJCN4YFQ5LFKOKSKQAZDCNJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent
Thanks, Brett. I understand why the behavior happens, I just don't understand the decision to implement imports this way. Since there's no warning in the documentation that removing items from sys.modules can break the fact that "import X.Y" defines "X.Y" (note that the "behind the curtain" stuff happens *before* the second import, so it's still the case that the second import does not define "X.Y" as implied by the docs), and there's also no warning that submodules must be removed at the same time as their parent, I would expect my example code to work. I don't see any downside to having "import X.Y" always set the Y attribute of X (instead of only setting it if 'X.Y' is not already in sys.modules), but if you think it's a bad idea, here's a suggestion for a paragraph to add at the end of PLR 5.4.2: "Note that the binding to the submodule object in the parent module's namespace is only added when the submodule is actually *loaded*. If the submodule is already present in `sys.modules` when it is imported (through any of the mechanisms above), then it will not be loaded again and no binding will be added to the parent module." If removing a module but not its submodules from sys.modules is considered "cheating" and could potentially break other parts of the import system, that should also be documented, e.g. by adding the sentence "If you delete a key for a module in `sys.modules`, you must also delete the keys for all submodules of that module." at the end of the 3rd paragraph of PLR 5.3.1. However, I would much rather not impose this restriction, since it seems unnecessarily restrictive (indeed, my code violates it but works fine, and changing it to transitively remove all submodules would necessitate reloading many modules which do not actually need to be reloaded). (Terry, thanks for your suggestion. My concern about adding such a vague warning is that to me, it reads as saying that all bets are off if you modify sys.modules by hand, which means it would never be safe to do so, i.e., the behavior might change arbitrarily in a future Python version. But in my opinion there are legitimate cases where it is necessary to ensure a module will be reloaded the next time it is imported, and the documented way to do that is to remove entries from sys.modules.) Daniel ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/N763W6AGD6NQ4IXVWMNGDL4DBN3LXBJ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Importing a submodule doesn't always set an attribute on its parent
On 4/9/2022 4:28 PM, Terry Reedy wrote: On 4/9/2022 5:09 AM, Arfrever Frehtes Taifersar Arahesis wrote: Not only deletion, but also random assignments: Ok. Change "Manual deletion of entries from sys.modules" to "Direct manipulation of sys.modules" I'm not sure it's worth the hassle to document this. There are no doubt hundreds of ways to break things. We can't enumerate all of them. Eric ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/WGP7SLEIEIDPJEOU3ANSRPGHWKLMM7QE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin, wrote: > So here's my proposal: > > - This API stays with the regular public API (Include/cpython/), but to > use it you'll need to #define Py_USING_UNSTABLE_API (name up for > bikeshedding). > I'm fine with the rest of what you suggest, but I don't think this is the right mechanical approach: * "unstable" is the wrong term. We already have an unstable API tier: the internal API, which can change even in maintenance releases. The value of the new tier is that it is "semi stable": stable in maintenance releases, unstable in feature releases. * the lesson I take from our stable ABI experience is that mixing two tiers of the API in a single header file is hard to maintain, as it's too easy to add a new API to the wrong section. A separate file that gets included automatically from the relevant header file(s) when the new definition is used makes the split much clearer. Cheers, Nick. > > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CTKKTHUV5R2A2RRN5DM32UQFNC42DDGJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin, wrote: > > This applies to: > > - Functions added in PEP 523 > - PyCode_New, PyCode_NewWithPosOnlyArgs > - Ideally anything documented as subject to change between minor > releases. (To be kind to users, if something is added later we should > again have one release of compiler warnings before requiring the opt-in. > Unless that API just changed and users would get errors anyway.) > Other candidate items for this tier: * non-opaque access to frame structs and any other key APIs needed to implement alternate eval loops with comparable performance to the default eval loop (unless & until we can figure out stable public APIs that can deliver equivalent performance) > * C APIs that provide access to compiled code whether in AST or opcode form (the API itself may be stable, but the compiled code isn't, so this is kinda covered by your last point) Cheers, Nick. > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CW543RWJU6EWAPTWCDDHEUS37UHSAU5I/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
On Fri, Apr 8, 2022 at 1:26 AM Malthe wrote: > This is an idea which has been brought up before, sometimes introduced > as "heresy". But an interesting twist has surfaced now which is > typing. > What for? To save a few keystrokes? Can't some IDE's add the import for you? Please don't drag Python in the direction of line noise. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/IOJ5J5OY4H2OLMXOGIX24EHGVT2VY5N4/ Code of Conduct: http://python.org/psf/codeofconduct/
