[Python-Dev] Re: Making code object APIs unstable
On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum wrote: > FWIW I've applied for an exception from the two-release deprecation policy > from the SC: > https://github.com/python/steering-council/issues/75 On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I didn't check how many are using Cython. yarl-1.6.3.tar.gz wsaccel-0.6.3.tar.gz wordcloud-1.8.1.tar.gz uvloop-0.16.0.tar.gz tslearn-0.5.2.tar.gz uamqp-1.4.1.tar.gz tweedledum-1.1.0.tar.gz tinycss-0.4.tar.gz thriftpy-0.3.9.tar.gz thriftpy2-0.4.14.tar.gz Theano-PyMC-1.1.2.tar.gz Theano-1.0.5.tar.gz TA-Lib-0.4.21.tar.gz tables-3.6.1.tar.gz ssh2-python-0.26.0.tar.gz srsly-2.4.1.tar.gz ssh-python-0.9.0.tar.gz statsmodels-0.12.2.tar.gz sphinx-gallery-0.9.0.tar.gz sktime-0.7.0.tar.gz Shapely-1.7.1.tar.gz wxPython-4.1.1.tar.gz scikit-image-0.18.2.tar.gz sasl-0.3.1.tar.gz scikit-surprise-1.1.1.tar.gz s2sphere-0.2.5.tar.gz ruptures-1.1.4.tar.gz runstats-2.0.0.tar.gz ruamel.yaml.clib-0.2.6.tar.gz reedsolo-1.5.4.tar.gz recordclass-0.15.1.tar.gz reportlab-3.6.1.tar.gz rasterio-1.2.6.tar.gz rapidfuzz-1.4.1.tar.gz qiskit-terra-0.18.1.tar.gz pyzmq-22.2.1.tar.gz pyxDamerauLevenshtein-1.7.0.tar.gz PyWavelets-1.1.1.tar.gz python-crfsuite-0.9.7.tar.gz py_spy-0.3.8.tar.gz pysimdjson-4.0.2.tar.gz pysam-0.16.0.1.tar.gz pypcap-1.2.3.tar.gz pyngrok-5.0.6.tar.gz PyLBFGS-0.2.0.13.tar.gz pyjq-2.5.2.tar.gz pyhacrf-datamade-0.2.5.tar.gz pyflux-0.4.15.tar.gz pygame-2.0.1.tar.gz pyemd-0.5.1.tar.gz pydevd-2.4.1.tar.gz pydevd-pycharm-212.5080.18.tar.gz plyvel-1.3.0.tar.gz pmdarima-1.8.2.tar.gz peewee-3.14.4.tar.gz paramiko-2.7.2.tar.gz osmium-3.2.0.tar.gz orderedset-2.0.3.tar.gz numpydoc-1.1.0.tar.gz numdifftools-0.9.40.tar.gz numba-0.53.1.tar.gz numcodecs-0.8.1.tar.gz NetfilterQueue-0.8.1.tar.gz neobolt-1.7.17.tar.gz Naked-0.1.31.tar.gz mypy-0.910.tar.gz msgpack-python-0.5.6.tar.gz msgpack-1.0.2.tar.gz mojimoji-0.0.11.tar.gz mpi4py-3.1.1.tar.gz matrixprofile-1.1.10.tar.gz marisa-trie-0.7.7.tar.gz lupa-1.9.tar.gz lxml-4.6.3.tar.gz lsm-db-0.6.4.tar.gz linearmodels-4.24.tar.gz lightfm-1.16.tar.gz Levenshtein-0.13.0.tar.gz leven-1.0.4.tar.gz lda-2.0.0.tar.gz jsonobject-0.9.10.tar.gz jq-1.2.1.tar.gz JPype1-1.3.0.tar.gz jenkspy-0.2.0.tar.gz implicit-0.4.4.tar.gz imgui-1.3.0.tar.gz imbalanced-learn-0.8.0.tar.gz imagecodecs-2021.7.30.tar.gz httptools-0.3.0.tar.gz httpretty-1.1.4.tar.gz hmmlearn-0.2.6.tar.gz hdbscan-0.8.27.tar.gz gssapi-1.6.14.tar.gz grpcio-tools-1.39.0.tar.gz grpcio-1.39.0.tar.gz graphene-federation-0.1.0.tar.gz GPy-1.10.0.tar.gz gluonnlp-0.10.0.tar.gz gevent-21.8.0.tar.gz gensim-4.0.1.tar.gz fuzzyset-0.0.19.tar.gz fuzzysearch-0.7.3.tar.gz frozendict-2.0.6.tar.gz flower-1.0.0.tar.gz Fiona-1.8.20.tar.gz fastrlock-0.6.tar.gz fastparquet-0.7.1.tar.gz fastdtw-0.3.4.tar.gz fastavro-1.4.4.tar.gz edlib-1.3.8.post2.tar.gz editdistance-0.5.3.tar.gz econml-0.12.0.tar.gz dtaidistance-2.3.2.tar.gz DoubleMetaphone-0.1.tar.gz django-localflavor-3.1.tar.gz dependency-injector-4.35.2.tar.gz dedupe-hcluster-0.3.8.tar.gz dedupe-2.0.8.tar.gz ddtrace-0.51.2.tar.gz cytoolz-0.11.0.tar.gz Cython-0.29.24.tar.gz correctionlib-2.0.0.tar.gz clickhouse-driver-0.2.1.tar.gz cityhash-0.2.3.post9.tar.gz cchardet-2.1.7.tar.gz causalml-0.11.1.tar.gz Cartopy-0.19.0.post1.tar.gz av-8.0.3.tar.gz asyncpg-0.24.0.tar.gz astral-2.2.tar.gz arch-5.0.1.tar.gz arcgis-1.9.0.tar.gz altgraph-0.17.tar.gz aiokafka-0.7.1.tar.gz aiohttp-3.7.4.post0.tar.gz affinegap-1.11.tar.gz Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6RO2WFU5Q7UQDVL72IRMT4T6L4GEAKB6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A better way to freeze modules
On Fri, Sep 3, 2021 at 5:32 AM Paul Moore wrote: > On Fri, 3 Sept 2021 at 10:29, Simon Cross > wrote: > > I think adding a meta path importer that reads from a standard > > optimized format could be a great addition. > > I think the biggest open question would be "what benefits does this > have over the existing zipimport?" +1 > > As you mentioned in your email, this is a big detour from the current > > start-up performance work, so I think practically the people working > > on performance are unlikely to take a detour from their detour right > > now. > > Agreed, it would probably have to be an independent development > initially. If it delivers better performance, then switching the > startup work to use it would give a second set of performance > improvements, which no-one is going to object to. Similarly, if it's > simpler to manage, then the maintainability benefits could justify > switching over. +1 > > * Write the meta path importer in a separate package (it sounds like > > you've already done a lot of the work and gained a lot of > > understanding of the issues while writing PyOxidizer!) > > This is the key thing, though. The import machinery allows new > importers to be written as standalone modules, so I'd strongly > recommend that the proposed format/importer gets developed as a PyPI > module initially, with the PEP then being simply a proposal that the > module gets added to the stdlib and/or built into the interpreter. FWIW, I'm a big fan of folks taking advantage of the flexibility of the import machinery and writing importers like this (especially ones that folks must explicitly enable). As noted elsewhere, it would need to prove its worth before we consider putting it into importlib. > The key argument would be bootstrapping, IMO. I would definitely expect > interest in something like this to be lower if it's an external module > (needing a dependency to load your other dependencies is suboptimal). > Conversely, though, if no-one shows any interest in a PyPI version of > this idea, that would strongly imply that it's not as useful in > practice as you'd hoped. Excellent point! > In particular, I'd involve the maintainers of pyinstaller in the > design. If a new "frozen module importer" mechanism isn't of interest > to them, it's probably not going to get the necessary support to be > worth adding to the stdlib. +1 > On a personal note, I love the flexibility of Python's import system, > and I've always wanted to write importers for additional storage > formats (import from a sqlite database, for instance). But I've never > actually done so, because a zipfile is basically always sufficient for > any practical use case I've had. One day I hope to find a real use > case, though :-) Cool! I'd love to see what you make. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YV2K3BPVDZRZTGLM4HWQEJWMVPI6BGHD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A better way to freeze modules
On Thu, Sep 2, 2021 at 10:46 PM Gregory Szorc wrote: > Over in https://bugs.python.org/issue45020 there is some exciting work around > expanding the use of the frozen importer to speed up Python interpreter > startup. I wholeheartedly support the effort and don't want to discourage > progress in this area. > > Simultaneously, I've been down this path before with PyOxidizer and feel like > I have some insight to share. Thanks for the support and for taking the time to share your insight! Your work on PyOxidizer is really neat. Before I dive in to replying, I want to be clear about what we are discussing here. There are two related topics: the impact of freezing stdlib modules and usability problems with frozen modules in general (stdlib or not). https://bugs.python.org/issue45020 is concerned with the former but prompted some good discussion about the latter. From what I understand, this python-dev thread is more about the latter (and then some). That's totally worth discussing! I just don't want the two topics to be unnecessarily conflated. FYI, frozen modules (effectively the .pyc data) are compiled into the Python binary and lhen loaded from there during import rather than from the filesystem. This allows us to avoid disk access, giving us a performance benefit, but we still have to unmarshal and execute the module code. It also allows us to have the import machinery written in pure Python (importlib._bootstrap and importlib._bootstrap_external). (Thanks Brett!) While frozen modules are derived from .py files, they currently have some differences from the corresponding source modules: the loader (which has less capability), the repr, frozen packages have __path__ set to [], and frozen modules don't have __file__, __cached__, etc. set. This has been the case for a long time. MAL worked on addressing __file__ but the effort stalled out. (See https://bugs.python.org/issue45020#msg400769 and especially https://bugs.python.org/issue21736.) The challenge with solving this for non-stdlib modules is that the frozen importer would need help to know where to find corresponding .py files. bpo-45020 is about freezing a small subset of the stdlib as a performance improvement. It's the 11 stdlib modules (plus encodings) that get imported every time during "./python -c pass". Freezing them provides a roughly 15% startup time improvement. (The 11 modules are: abc, codecs, encodings, io, _collections_abc, _site_builtins, os, os.path, genericpath, site, and stat. Maybe there are a few other modules it would make sense to freeze but we're starting with those 11.) This work is probably somewhat affected by the differences between frozen and source modules, and we may need to set an appropriate __file__ on frozen stdlib modules to avoid impacting folks that expect any of those stdlib modules to have it set. Otherwise, for bpo-45020 there likely isn't much more we need to do about frozen stdlib modules shipping with CPython by default. Regardless, bpo-45020 doesn't introduce any new problems; rather it slightly exposes the existing ones. In contrast to the use of frozen modules in default Python builds, there are a number of tools in the community for freezing modules (both stdlib and not) into custom Python binaries, like PyOxidizer and MAL's PyRun. Such tools would benefit from broader compatibility between frozen modules and the corresponding source modules. Consequently the tool maintainers would be the most likely drivers of any effort to improve frozen modules (which the discussion with MAL and Gregory bears out). The tools would especially benefit if those improvements could apply to non-stdlib modules, which requires a more complex solution than is needed for stdlib modules. At the (relative) extreme is to throw out the existing frozen module approach (or even the "unmarshal + exec" approach of source-based modules) and replace it with something more efficient and/or more compatible (and cross-platform). From what I understood, this is the main focus of this thread. It's interesting stuff and I hope the discussion renders a productive result. FTR, in bpo-45020 Gregory helpfully linked to some insightful material related to PyOxidizer and frozen modules: * https://github.com/indygreg/PyOxidizer/issues/69 * https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html?highlight=__file__#file-and-cached-module-attributes * https://pypi.org/project/oxidized-importer/ and https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html With that said, on to replying. :) > I don't think I'll be offending anyone by saying the existing CPython frozen > importer is quite primitive in terms of functionality: it does the minimum it > needs to do to support importing module bytecode embedded in the interpreter > binary [for purposes of bootstrapping the Python-based importlib modules]. > The C struct representing frozen modules is literally just the
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2021-08-27 - 2021-09-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7409 ( -4) closed 49419 (+67) total 56828 (+63) Open issues with patches: 2945 Issues opened (38) == #20658: os.environ.clear() fails with empty keys (posix.unsetenv) https://bugs.python.org/issue20658 reopened by iritkatriel #37330: open(): remove 'U' mode, deprecated since Python 3.3 https://bugs.python.org/issue37330 reopened by vstinner #45033: Calls to PyErr_PrintEx in destructors cause calling async func https://bugs.python.org/issue45033 opened by zbentley #45034: Improve struct.pack out of range error messages https://bugs.python.org/issue45034 opened by steven.daprano #45035: sysconfig's posix_home scheme has different platlib value to d https://bugs.python.org/issue45035 opened by uranusjr #45036: turtle.onrelease() event doesn't get triggered sometimes https://bugs.python.org/issue45036 opened by techn010je11y #45040: [sqlite3] optimise transaction control functions https://bugs.python.org/issue45040 opened by erlendaasland #45041: [sqlite3] simplify executescript() https://bugs.python.org/issue45041 opened by erlendaasland #45042: Many multiprocessing tests are silently skipped since 3.9 https://bugs.python.org/issue45042 opened by serhiy.storchaka #45046: Add support of context managers in unittest https://bugs.python.org/issue45046 opened by serhiy.storchaka #45052: WithProcessesTestSharedMemory.test_shared_memory_basics fails https://bugs.python.org/issue45052 opened by sobolevn #45053: MD5SumTests.test_checksum_fodder fails on Windows https://bugs.python.org/issue45053 opened by sobolevn #45054: json module should issue warning about duplicate keys https://bugs.python.org/issue45054 opened by Zeturic #45055: Fresh build on Windows fails the first time for zlib.c https://bugs.python.org/issue45055 opened by gvanrossum #45056: compiler: Unnecessary None in co_consts https://bugs.python.org/issue45056 opened by methane #45058: Undefined behavior for syntax "except AError or BError:" accep https://bugs.python.org/issue45058 opened by kftse20031207 #45060: Do not use the equality operators with None https://bugs.python.org/issue45060 opened by serhiy.storchaka #45062: test_asyncio: test_huge_content_recvinto() failed on PPC64LE R https://bugs.python.org/issue45062 opened by vstinner #45063: PEP 657 Fine Grained Error Locations: omit indicators if they https://bugs.python.org/issue45063 opened by vstinner #45065: test_asyncio failed (env changed) on s390x RHEL8 Refleaks 3.10 https://bugs.python.org/issue45065 opened by vstinner #45066: email parser fails to decode quoted-printable rfc822 message a https://bugs.python.org/issue45066 opened by anarcat #45067: Failed to build _curses on CentOS 7 https://bugs.python.org/issue45067 opened by orsenthil #45069: python 3.9.2 contains libcrypto-1_1.dll and libssl-1_1.dll ass https://bugs.python.org/issue45069 opened by xcl123 #45073: windows installer quiet installation targetdir escapes "quote" https://bugs.python.org/issue45073 opened by DMI-1407 #45074: asyncio hang in subprocess wait_closed() on Windows, BrokenPip https://bugs.python.org/issue45074 opened by byllyfish #45075: confusion between frame and frame_summary in traceback module https://bugs.python.org/issue45075 opened by iritkatriel #45077: multiprocessing.Pool(64) crashes on Windows https://bugs.python.org/issue45077 opened by saschanaz #45078: test_importlib: test_read_bytes() fails on AMD64 Windows8.1 No https://bugs.python.org/issue45078 opened by vstinner #45083: Need to use the exception class qualname when rendering except https://bugs.python.org/issue45083 opened by iritkatriel #45084: urllib.parse: remove deprecated functions (splittype, to_bytes https://bugs.python.org/issue45084 opened by vstinner #45086: f-string unmatched ']' https://bugs.python.org/issue45086 opened by Greg Kuhn #45088: Coroutines & async generators disagree on the iteration protoc https://bugs.python.org/issue45088 opened by yselivanov #45089: [sqlite3] the trace callback does not raise exceptions on erro https://bugs.python.org/issue45089 opened by erlendaasland #45090: Add pairwise to What's New in Python 3.10; mark it as new in i https://bugs.python.org/issue45090 opened by ramalho #45091: inspect.Parameter.__str__ does not include subscripted types i https://bugs.python.org/issue45091 opened by antonio-caceres #45093: Add method to compare dicts accounting for order https://bugs.python.org/issue45093 opened by mcarans #45094: Consider using __forceinline and __attribute__((always_inline) https://bugs.python.org/issue45094 opened by vstinner #45095: Easier loggers traversal tree with a logger.getChildren method https://bugs.python.org/issue45095 opened by sblondon Most
[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries
> > As part of PEP 588, migrating bugs.python.org issues to Github, there > are two current mailing list related items that need a replacement or > need to be turned down. > > 1. Weekly summary emails with bug counts and issues from the week, > example: > https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/ > > 2. Emails sent to the new-bugs-announce and python-bugs-list for new > issues and comments to existing issues. > I occasionally rely on these. Those weren't noise to me. I assume new instructure might bring other benefits, so we won't be certain if we will need these new-bugs-announcements emails. Or those might be replaced with something better. Thank you, Senthil ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OARW5EZTBABBAZLBS2U3FX3RXKJSETFR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries
On Mon, Aug 23, 2021 at 4:16 PM Ammar Askar wrote: > As part of PEP 588, migrating bugs.python.org issues to Github, Thanks for working on this! > 1. Weekly summary emails with bug counts and issues from the week, > 2. Emails sent to the new-bugs-announce and python-bugs-list for new I rely on both these. They help improve signal-to-noise and make it easier to quickly get back up-to-date if I'm out for a while. -eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IZJQIDKKL7KBGPUEL2YQO44FZIMLTZPO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A better way to freeze modules
On Fri, 3 Sept 2021 at 10:29, Simon Cross wrote: > > Hi Gregory, > > I think adding a meta path importer that reads from a standard > optimized format could be a great addition. I think the biggest open question would be "what benefits does this have over the existing zipimport?" Maybe it could be a little faster? But would the downside of it not being possible to manage the format with existing standard tools outweigh that? A clear description of how to decide which is the most appropriate to use in a given situation between the new format and a zipfile would be a benefit here. > As you mentioned in your email, this is a big detour from the current > start-up performance work, so I think practically the people working > on performance are unlikely to take a detour from their detour right > now. Agreed, it would probably have to be an independent development initially. If it delivers better performance, then switching the startup work to use it would give a second set of performance improvements, which no-one is going to object to. Similarly, if it's simpler to manage, then the maintainability benefits could justify switching over. > * Ask if there are any Python core developers who would be willing to > look at the early stages of the code and/or PEP that you might produce > in the next couple of steps. Perhaps also ask on one of the packaging > mailing lists. If you get others involved as reviewers or contributors > from the start, convincing them later that it is a good idea will be > much easier. :) I'd be willing to look. I'm more interested in the design at this stage than in looking at code, as it's awfully easy to develop something that ends up being a "solution looking for a problem", so a solid case for having a general solution would be important for me. > * Write the meta path importer in a separate package (it sounds like > you've already done a lot of the work and gained a lot of > understanding of the issues while writing PyOxidizer!) This is the key thing, though. The import machinery allows new importers to be written as standalone modules, so I'd strongly recommend that the proposed format/importer gets developed as a PyPI module initially, with the PEP then being simply a proposal that the module gets added to the stdlib and/or built into the interpreter. The key argument would be bootstrapping, IMO. I would definitely expect interest in something like this to be lower if it's an external module (needing a dependency to load your other dependencies is suboptimal). Conversely, though, if no-one shows any interest in a PyPI version of this idea, that would strongly imply that it's not as useful in practice as you'd hoped. In particular, I'd involve the maintainers of pyinstaller in the design. If a new "frozen module importer" mechanism isn't of interest to them, it's probably not going to get the necessary support to be worth adding to the stdlib. > * Write a PEP. > > It seems to me that PEPs that come with an implementation and the > support of a few existing core developers have a much less painful PEP > review process. Agreed. In particular, existing code with a clearly demonstrated user base only has to persuade people that being in the core is important. Most proposals that I've seen which could be developed as a PyPI module never get anywhere because it turns out no-one is willing to do the work. You don't need to be a core developer to write a PyPI module, so if no-one has done that, it's likely to be either because the implementation needs tight integration into the core, or because nobody is actually as interested in the issue as you thought... On a personal note, I love the flexibility of Python's import system, and I've always wanted to write importers for additional storage formats (import from a sqlite database, for instance). But I've never actually done so, because a zipfile is basically always sufficient for any practical use case I've had. One day I hope to find a real use case, though :-) > Thank you for writing PyOxidizer and offering some of your time to > help make Python itself better. +1 Paul. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2THOX327G2SBUBNC4CEFK2JOH7VICFHC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A better way to freeze modules
Hi Gregory, I think adding a meta path importer that reads from a standard optimized format could be a great addition. As you mentioned in your email, this is a big detour from the current start-up performance work, so I think practically the people working on performance are unlikely to take a detour from their detour right now. If you would like to see your suggested feature in Python, I *think* the following might be a reasonable approach: * Email python-dev about your idea (done already! :) * Ask if there are any Python core developers who would be willing to look at the early stages of the code and/or PEP that you might produce in the next couple of steps. Perhaps also ask on one of the packaging mailing lists. If you get others involved as reviewers or contributors from the start, convincing them later that it is a good idea will be much easier. :) * Write the meta path importer in a separate package (it sounds like you've already done a lot of the work and gained a lot of understanding of the issues while writing PyOxidizer!) * Write a PEP. It seems to me that PEPs that come with an implementation and the support of a few existing core developers have a much less painful PEP review process. Thank you for writing PyOxidizer and offering some of your time to help make Python itself better. Yours sincerely, Simon Cross P.S. I am not a core developer, and I haven't even written any PEPs. :) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LMHQ2S2YDJHQNQG3U65GIUPU6IB5QDXY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Discrepancy between what aiter() and `async for` requires on purpose?
I think the C implementation of PyAiter_Check was a translation of the Python code `isinstance(..., collections.abc.AsyncIterator)`, but I agree that it would be more consistent to just check for __anext__. There were comments at the time here: https://github.com/python/cpython/pull/8895#discussion_r532833905 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VLT475NI2ORNSHFZZVZVD4QYOPG66SQK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: A better way to freeze modules
My quick reaction was somewhat different - it would be a great idea, but it’s entirely possible to implement this outside the stdlib as a 3rd party module. So the fact that no-one has yet done so means there’s less general interest than the OP is suggesting. And from my experience, the reason for that is that zipimport is almost always sufficient. That’s what tools like pyinstaller use, for example. Paul On Fri, 3 Sep 2021 at 06:25, Guido van Rossum wrote: > Quick reaction: This feels like a bait and switch to me. Also, there are > many advantages to using a standard format like zip (many formats are > really zip with some conventions). Finally, the bytecode format you are > using is “marshal”, and is fully portable — as is zip. > > On Thu, Sep 2, 2021 at 21:44 Gregory Szorc > wrote: > >> Over in https://bugs.python.org/issue45020 there is some exciting work >> around expanding the use of the frozen importer to speed up Python >> interpreter startup. I wholeheartedly support the effort and don't want to >> discourage progress in this area. >> >> Simultaneously, I've been down this path before with PyOxidizer and feel >> like I have some insight to share. >> >> I don't think I'll be offending anyone by saying the existing CPython >> frozen importer is quite primitive in terms of functionality: it does the >> minimum it needs to do to support importing module bytecode embedded in the >> interpreter binary [for purposes of bootstrapping the Python-based >> importlib modules]. The C struct representing frozen modules is literally >> just the module name and a pointer to a sized buffer containing bytecode. >> >> In issue45020 there is talk of enhancing the functionality of the frozen >> importer to support its potential broader use. For example, setting >> __file__ or exposing .__loader__.get_source(). I support the overall >> initiative. >> >> However, introducing enhanced functionality of the frozen importer will >> at the C level require either: >> >> a) backwards incompatible changes to the C API to support additional >> metadata on frozen modules (or at the very least a supplementary API that >> fragments what a "frozen" module is). >> b) CPython only hacks to support additional functionality for "freezing" >> the standard library for purposes of speeding up startup. >> >> I'm not a CPython core developer, but neither "a" nor "b" seem ideal to >> me. "a" is backwards incompatible. "b" seems like a stop-gap solution until >> a more generic version is available outside the CPython standard library. >> >> From my experience with PyOxidizer and software in general, here is what >> I think is going to happen: >> >> 1. CPython enhances the frozen importer to be usable in more situations. >> 2. Python programmers realize this solution has performance and >> ease-of-distribution wins and want to use it more. >> 3. Limitations in the frozen importer are found. Bugs are reported. >> Feature requests are made. >> 4. The frozen importer keeps getting incrementally extended or Python >> developers grow frustrated that its enhancements are only available to the >> standard library. You end up slowly reimplementing the importing mechanism >> in C (remember Python 2?) or disappoint users. >> >> Rather than extending the frozen importer, I would suggest considering an >> alternative solution that is far more useful to the long-term success of >> Python: I would consider building a fully-featured, generic importer that >> is capable of importing modules and resource data from a well-defined and >> portable serialization format / data structure that isn't defined by C >> structs and APIs. >> >> Instead of defining module bytecode (and possible additional minimal >> metadata) in C structs in a frozen modules array (or an equivalent C API), >> what if we instead defined a serialization format for representing the >> contents of loadable Python data (module source, module bytecode, resource >> files, extension module library data, etc)? We could then point the Python >> interpreter at instances of this data structure (in memory or in files) so >> it could import/load the resources within using a meta path importer. >> >> What if this serialization format were designed so that it was extremely >> efficient to parse and imports could be serviced with the same trivially >> minimal overhead that the frozen importer currently has? We could embed >> these data structures in produced binaries and achieve the same desirable >> results we'll be getting in issue45020 all while delivering a more generic >> solution. >> >> What if this serialization format were portable across machines? The >> entire Python ecosystem could leverage it as a container format for >> distributing Python resources. Rather than splatting dozens or hundreds of >> files on the filesystem, you could write a single file with all of a >> package's resources. Bugs around filesystem implementation details such as >> case (in)sensitivity and Unicode normalization
