[issue45020] Freeze all modules imported during startup.

[Christian Heimes]

Christian Heimes  added the comment:


New changeset 5c4b19ec49a5fbad65a682225f7cfed8b78f2a2f by Christian Heimes in 
branch 'main':
bpo-45020: Fix strict-prototypes warning (GH-29755)
https://github.com/python/cpython/commit/5c4b19ec49a5fbad65a682225f7cfed8b78f2a2f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Christian Heimes]

Change by Christian Heimes :


--
nosy: +christian.heimes, christian.heimes, christian.heimes
nosy_count: 24.0 -> 25.0
pull_requests: +27992, 27993, 27994
pull_request: https://github.com/python/cpython/pull/29755

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Christian Heimes]

Change by Christian Heimes :


--
nosy: +christian.heimes, christian.heimes
nosy_count: 24.0 -> 25.0
pull_requests: +27992, 27993
pull_request: https://github.com/python/cpython/pull/29755

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Christian Heimes]

Change by Christian Heimes :


--
nosy: +christian.heimes
nosy_count: 24.0 -> 25.0
pull_requests: +27992
pull_request: https://github.com/python/cpython/pull/29755

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:


New changeset 1cbaa505d007e11c4a1f0d2073d72b6c02c7147c by Guido van Rossum in 
branch 'main':
bpo-45696: Deep-freeze selected modules (GH-29118)
https://github.com/python/cpython/commit/1cbaa505d007e11c4a1f0d2073d72b6c02c7147c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

I consider this done.  There is some lingering follow-up work, for which I've 
created a number of issues:

* https://bugs.python.org/issue45396 - -X frozen_modules=off ignores custom 
frozen modules
* https://bugs.python.org/issue45395 - custom modules override frozen stdlib 
modules
* https://bugs.python.org/issue45651 - frozen_modules default not right on 
Windows
* https://bugs.python.org/issue45272 - "os.path" should be in the list of 
frozen modules
* https://bugs.python.org/issue45653 - freeze encodings
* https://bugs.python.org/issue45654 - freeze runpy
* https://bugs.python.org/issue45660 - freeze argparse
* https://bugs.python.org/issue45661 - freeze other stdlib modules
* https://bugs.python.org/issue45273 - os-specific modules frozen unnecessarily?
* https://bugs.python.org/issue45096 - update Tools/freeze
* https://bugs.python.org/issue45379 - improving frozen-related errors
* https://github.com/python/devguide/issues/756 - add devguide info about 
frozen modules

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum  wrote:
> Is gh-28940 only for UNIX?
> I built on Windows with default options (PCbuild\build.bat) and it looks like 
> the frozen modules are used by default even though I am running in the source 
> directory. (I put a printf() call in unmarshal_frozen_code().)

FYI, I opened https://bugs.python.org/issue45651 for sorting this out.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 6afb285ff0790471a6858e44f85d143f07fda70c by Eric Snow in branch 
'main':
bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997)
https://github.com/python/cpython/commit/6afb285ff0790471a6858e44f85d143f07fda70c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Filipe Laíns]

Change by Filipe Laíns :


--
nosy: +FFY00

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum  wrote:
> Is gh-28940 only for UNIX?

It wasn't meant to be. :(

> I built on Windows with default options (PCbuild\build.bat) and it looks like 
> the frozen modules are used by default even though I am running in the source 
> directory. (I put a printf() call in unmarshal_frozen_code().)

I'll look into this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

Is gh-28940 only for UNIX?

I built on Windows with default options (PCbuild\build.bat) and it looks like 
the frozen modules are used by default even though I am running in the source 
directory. (I put a printf() call in unmarshal_frozen_code().)

I also put a printf() in is_dev_env() and found that it returns 0 on this check:

/* If dirname() is the same for both then it is a dev build. */
if (len != _Py_find_basename(stdlib)) {
return 0;
}

I assume that's because the binary (in my case at least) is at 
PCbuild\amd64\python.exe which is not the same as my current directory (which 
is the repo root).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Gregory P. Smith]

Gregory P. Smith  added the comment:

could changes related to this be the cause of 
https://bugs.python.org/issue45506 ?

out of tree builds in main usually cannot pass key tests today.  they often 
hang or blow up with strange exceptions.

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b by Eric Snow in branch 
'main':
bpo-45020: Default to using frozen modules unless running from source tree. 
(gh-28940)
https://github.com/python/cpython/commit/b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27280
pull_request: https://github.com/python/cpython/pull/28997

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27229
pull_request: https://github.com/python/cpython/pull/28940

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Tue, Oct 5, 2021 at 11:31 AM Guido van Rossum  wrote:
> Whoa. os.path is not always an alias for posixpath, is it?

Steve brought this to my attention a couple weeks ago.  Bottom line:
the frozen module entry is only there for checks, not for actual
import, but should probably be removed regardless.  See
https://bugs.python.org/issue45272.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Larry Hastings]

Larry Hastings  added the comment:

Nope.  On Windows, os.path is "ntpath".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

Whoa. os.path is not always an alias for posixpath, is it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 08285d563e64c179a56ab2f952345b3dbcdb54f3 by Eric Snow in branch 
'main':
bpo-45020: Identify which frozen modules are actually aliases. (gh-28655)
https://github.com/python/cpython/commit/08285d563e64c179a56ab2f952345b3dbcdb54f3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

And it's most definitely 3.11, not 3.9. (Did you mean to change a different 
issue?)

--
versions: +Python 3.11 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

@santhu_reddy12, why did you assign this to the Parser category? IMO this issue 
is clearly in the Build category. (We haven't met, I assume you have triage 
permissions?)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[santhosh]

Change by santhosh :


--
components: +Parser -Build
nosy: +lys.nikolaou, pablogsal, santhu_reddy12
versions: +Python 3.9 -Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27045
pull_request: https://github.com/python/cpython/pull/28655

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27030
pull_request: https://github.com/python/cpython/pull/28665

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 7e5c107541726b90d3f2e6e69ef37180cf58335d by Eric Snow in branch 
'main':
bpo-45020: Add more test cases for frozen modules. (gh-28664)
https://github.com/python/cpython/commit/7e5c107541726b90d3f2e6e69ef37180cf58335d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27029
pull_request: https://github.com/python/cpython/pull/28664

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 45ca1c04139300ec0289a32f78c7ac922a4f7b07 by Eric Snow in branch 
'main':
bpo-45020: Do not freeze /__init__.py twice. (gh-28635)
https://github.com/python/cpython/commit/45ca1c04139300ec0289a32f78c7ac922a4f7b07


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +27004
pull_request: https://github.com/python/cpython/pull/28635

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26972
pull_request: https://github.com/python/cpython/pull/28590

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26965
pull_request: https://github.com/python/cpython/pull/28583

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 25.09.2021 18:20, STINNER Victor wrote:
> 
> STINNER Victor  added the comment:
> 
> Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c 
> and PC/getpathp.c.

Now that you mention it: yes, that as well :-) But os.py is used in the
Python stdlib code as well, just search for "os.__file__" to see a few
such uses.

If you search for ".__file__" you'll find that there are quite a few
cases in the test suite expecting that attribute on other stdlib modules
as well. The attribute may officially be optional, but in reality a
lot of code expects to find it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:

Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c 
and PC/getpathp.c.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

The plan is to add __file__ back, based on where the stdlib lives (we won’t do 
any stat() calls for frozen files).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

Eric, I noticed that you are freezing os.py. Please be aware that
this module is often being used as indicator for where the stdlib
was installed (the stdlib itself does this in site.py to read the
LICENSE and the test suite also uses os.__file__ in a couple of
places).

It may be worth changing the stdlib to pick a different module
as landmark for this purpose.

Also: Unless you have added the .__file__ attribute to frozen
modules, much of this landmark checking code will fail... which is
the reason I added the attribute to frozen modules in PyRun.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26937
pull_request: https://github.com/python/cpython/pull/28554

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 7c801e0fa603b155eab3fd19698aa90854ac5a7b by Eric Snow in branch 
'main':
bpo-45020: Fix some corner cases for frozen module generation. (gh-28538)
https://github.com/python/cpython/commit/7c801e0fa603b155eab3fd19698aa90854ac5a7b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26923
pull_request: https://github.com/python/cpython/pull/28538

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Gregory Szorc]

Gregory Szorc  added the comment:

I'll throw out that __file__ is strictly optional, per 
https://docs.python.org/3/reference/datamodel.html. IMO it is more important to 
define importlib.abc.InspectLoader than __file__ (so tracebacks have source 
context).

Given that __file__ is optional, I'll throw out the idea of shipping modules 
without __file__/__cached__ and see what issues/pushback occurs. I anticipate 
this will likely result in having to restore advertising __file__ in frozen 
modules in 3.11 due to popular demand. But we /could/ use this as an 
opportunity to start attempting to ween people off __file__ and onto more 
appropriate APIs, like importlib.resources. (importlib.resources was introduced 
in 3.7 and I'm not sure that's old enough to force people onto without relying 
on a shim like https://pypi.org/project/importlib-resources/.)

Alternatively, defining a path hook referencing the python executable that 
knows how to service frozen modules could also work. zipimporter does this 
(__file__ is /) and it is surprisingly not as 
brittle as one would think. It might be a good middle ground.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 22.09.2021 20:47, Brett Cannon wrote:
> What about if there isn't a pre-computed location for __file__? I could 
> imagine a self-contained CPython build where there is no concept of a file 
> location on disk for anything using this.

This does work and is enough to make most code out there happy.

I use e.g. "/os.py" in PyRun. There is no os.py file to load,
but tracebacks and inspection tools work just fine with this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Brett Cannon]

Brett Cannon  added the comment:

What about if there isn't a pre-computed location for __file__? I could imagine 
a self-contained CPython build where there is no concept of a file location on 
disk for anything using this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

I'm still torn about the need for a __file__ attribute. Assuming it's more 
likely that people write code that *reads* stdlib source for some reason (maybe 
printing more context for error messages) than *writing* it (which would be 
truly strange), the amount of 3rd party code that might break due to the lack 
of __file__ can be expected to be larger than the amount of code (or number of 
users) that would be confused to having a __file__ pointing to a file that 
isn't actually read by the interpreter.

Okay, so I'm no longer torn. We should set __file__ for frozen modules based on 
the pre-computed location of the stdlib.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:

runpy startup time matters a lot for "python3 -m module" startup  time!

In Python 3.10 (bpo-41006 and bpo-41718), I reduced the number of modules 
imported by runpy:

"The runpy module now imports fewer modules. The python3 -m module-name command 
startup time is 1.4x faster in average. On Linux, python3 -I -m module-name 
imports 69 modules on Python 3.9, whereas it only imports 51 modules (-18) on 
Python 3.10. (Contributed by Victor Stinner in bpo-41006 and bpo-41718.)"

https://docs.python.org/dev/whatsnew/3.10.html#optimizations

--

For argparse, maybe it could use a few lazy imports to reduce the number of 
indirect impots on "import argparse"?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Raymond Hettinger]

Raymond Hettinger  added the comment:

> Raymond, do you think we should also freeze the dependencies
> of runpy (so "python -m " also starts faster)?

Yes, please.  

The '-m' load option can be considered core startup functionality.  It is often 
the recommended way to launch command line tools outside of a virtual 
environment.

   python -m pip install some_package

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

> It would be nice to freeze argparse.py and its dependencies.  For 
> command-line tools, startup time is important.

I quickly checked, and argparse has at least these dependencies:

argparse
re
enum
types
operator
functools
collections
keyword
reprlib
sre_compile
sre_parse
sre_constants
copyreg
gettext

Raymond, do you think we should also freeze the dependencies of runpy (so 
"python -m " also starts faster)?

That would be

runpy
importlib
warnings
importlib.machinery
importlib.util
importlib._abc
contextlib
collections
keyword
operator
reprlib
functools
types

(I didn't dedupe this from the previous list.)

With all these modules frozen we'd probably run more risk of two things:

- frozen modules don't have a __file__ (and are missing a few other, less 
commonly used, attributes)
- editing a frozen module requires rebuilding, or running with -X 
frozen_modules=off

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Raymond Hettinger]

Raymond Hettinger  added the comment:

It would be nice to freeze argparse.py and its dependencies.  For command-line 
tools, startup time is important.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Fri, Sep 17, 2021 at 4:22 PM Guido van Rossum  wrote:
> BTW, why does the script still run Programs/_freeze_module over all the 
> modules to be frozen? Isn't that up to the Makefile or its Windows equivalent?

Yeah, it used to matter but we probably don't need to do that any longer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 090591636c4f03ce06a039079bd7716a5b23631e by Eric Snow in branch 
'main':
bpo-45020: Freeze os, site, and codecs. (gh-28398)
https://github.com/python/cpython/commit/090591636c4f03ce06a039079bd7716a5b23631e


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

BTW, why does the script still run Programs/_freeze_module over all the modules 
to be frozen? Isn't that up to the Makefile or its Windows equivalent?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

(Ah, now I recall how the $(srcdir) mechanism works again. Generated files do 
*not* have a $(srcdir) prefix, since they are generated in the destination 
directory. Hence the fix for the buildbot failure a few days ago. I take back 
my skepticism about that.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

Thanks for fixing that, Victor!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 41551ee7e24fb6c58846836d3655dbb212281206 by Victor Stinner in 
branch 'main':
bpo-45020: Fix build out of source tree (GH-28410)
https://github.com/python/cpython/commit/41551ee7e24fb6c58846836d3655dbb212281206


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +26822
pull_request: https://github.com/python/cpython/pull/28410

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:

There is also a minor issue, "make regen-frozen" changes the end of line of two 
PCbuild/ files: see bpo-45231.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:

It's no longer possible to build Python from a different directory. Example:

git clean -fdx
mkdir build
cd build/
../configure --with-pydebug
make

It fails with:

../Programs/_freeze_module importlib._bootstrap ../Lib/importlib/_bootstrap.py 
../Python/frozen_modules/importlib__bootstrap.h
make: ../Programs/_freeze_module: No such file or directory

I'm working on a fix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26812
pull_request: https://github.com/python/cpython/pull/28398

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset fdc6b3d9316501d2f0068a1bf4334debc1949e62 by Eric Snow in branch 
'main':
bpo-45020: Drop the frozen .h files from the repo. (gh-28392)
https://github.com/python/cpython/commit/fdc6b3d9316501d2f0068a1bf4334debc1949e62


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26804
pull_request: https://github.com/python/cpython/pull/28392

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

I left a comment on bpo-45220.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[STINNER Victor]

STINNER Victor  added the comment:

Is bpo-45220 "Windows builds sometimes fail on Azure and GitHub Action: fatal 
error RC1116: RC terminating after preprocessor errors" related to this issue?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset 9fd87a5fe5c468cf94265365091267838b004b7f by Eric Snow in branch 
'main':
bpo-45020: Revert "Drop the frozen .h files from the repo." (gh-28380)
https://github.com/python/cpython/commit/9fd87a5fe5c468cf94265365091267838b004b7f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26795
pull_request: https://github.com/python/cpython/pull/28380

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

Never mind.  It's pretty late here so I'm going to revert it and sort it out in 
the morning.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

FWIW, Python/frozen_modules/__hello__.h is still in the repo somehow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

Looks like that last commit broke one of the buildbots:

https://buildbot.python.org/all/#/builders/483/builds/812

I'll fix that right away.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset a9757bf34d8b4cb3c24bbb70d50a06c815e2e8f3 by Eric Snow in branch 
'main':
bpo-45020: Drop the frozen .h files from the repo. (gh-28375)
https://github.com/python/cpython/commit/a9757bf34d8b4cb3c24bbb70d50a06c815e2e8f3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Wed, Sep 15, 2021 at 12:03 PM Guido van Rossum
 wrote:
> I would move "default to "on" (except if actually running out of the source 
> tree)" to the "maybe" category. I left a few comments in other deps. I think 
> we should start by turning this on by default in PGO builds.

Sounds good.

> Separately, I encourage you to collect reliable performance numbers. It would 
> be nice to see a dip on speed.python.org for this benchmark:

I'll do that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26790
pull_request: https://github.com/python/cpython/pull/28375

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

I would move "default to "on" (except if actually running out of the source 
tree)" to the "maybe" category. I left a few comments in other deps. I think we 
should start by turning this on by default in PGO builds.

Separately, I encourage you to collect reliable performance numbers. It would 
be nice to see a dip on speed.python.org for this benchmark:

https://speed.python.org/timeline/#/?exe=12=python_startup=1=50=off=on=on

(but that won't show up until we turn this on by default for PGO builds).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

At this point the fundamental work is done.  Here are some follow-up tasks to 
wrap up this issue:

* freeze the remaining stdlib modules imported during startup (os, site, 
codecs, encodings.*)
   + blocked by bpo-45186 and bpo-45188
* default to "on" (except if actually running out of the source tree)
   + blocked by bpo-45211 (if we want to minimize disk access)
* always default to "on" if it's a PGO build, even if running out of the source 
tree
* stop tracking the frozen module .h files in the repo
   + blocked by bpo-45188
* (maybe) freeze modules imported for "python -m ..." (e.g. runpy)
* (maybe) freeze a small subset of the encodings (e.g. UTF-8)
   + blocked by a valid __path__ (see below) and consequently bpo-45211

Other related follow-up tasks:

* (maybe) make frozen stdlib modules more like source modules, with __file__ 
and __path__ (see bpo-21736)
   + blocked by bpo-45211
   + __path__ allows us to have frozen stdlib packages with non-frozen 
submodules (e.g. encodings)
* (maybe) freeze other modules (e.g. commonly used modules, sysconfig)
* (maybe) use something other than linear search for looking up frozen modules 
(see bpo-45213)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset cbeb81971057d6c382f45ecce92df2b204d4106a by Eric Snow in branch 
'main':
bpo-45020: Freeze some of the modules imported during startup. (gh-28335)
https://github.com/python/cpython/commit/cbeb81971057d6c382f45ecce92df2b204d4106a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[miss-islington]

miss-islington  added the comment:


New changeset f71b86e0ae194613d235086755c6a44266978be1 by Miss Islington (bot) 
in branch '3.9':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/f71b86e0ae194613d235086755c6a44266978be1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[miss-islington]

miss-islington  added the comment:


New changeset 8a9396cf1d9e1ce558841095e1ce0d3c23b7a8aa by Miss Islington (bot) 
in branch '3.10':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/8a9396cf1d9e1ce558841095e1ce0d3c23b7a8aa


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 15.0 -> 16.0
pull_requests: +26758
pull_request: https://github.com/python/cpython/pull/28345

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[miss-islington]

Change by miss-islington :


--
pull_requests: +26759
pull_request: https://github.com/python/cpython/pull/28346

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Terry J. Reedy]

Terry J. Reedy  added the comment:


New changeset 369bf949ccbb689cd4638b29b4c0c12db79b927c by Terry Jan Reedy in 
branch 'main':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/369bf949ccbb689cd4638b29b4c0c12db79b927c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Terry J. Reedy]

Change by Terry J. Reedy :


--
nosy: +terry.reedy
nosy_count: 14.0 -> 15.0
pull_requests: +26756
pull_request: https://github.com/python/cpython/pull/28344

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:


New changeset a65c86889e208dddb26a7ebe7840c24edbcca775 by Eric Snow in branch 
'main':
bpo-45020: Add -X frozen_modules=[on|off] to explicitly control use of frozen 
modules. (gh-28320)
https://github.com/python/cpython/commit/a65c86889e208dddb26a7ebe7840c24edbcca775


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26745
pull_request: https://github.com/python/cpython/pull/28335

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Mon, Sep 13, 2021 at 2:59 PM Guido van Rossum  wrote:
> If you reduce the number of modules being frozen you could probably manage
> to land this (or most of it) before tackling those other issues.

Yeah, that's what I'm doing. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +26729
pull_request: https://github.com/python/cpython/pull/28320

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

If you reduce the number of modules being frozen you could probably manage
to land this (or most of it) before tackling those other issues.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
dependencies: +De-couple the Windows builds from freezing modules., Marshal 
output isn't completely deterministic.

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

FWIW, I've not found the importer for frozen modules to be lacking
features. When using frozen modules, you don't expect to see source
code, so the whole part about finding source code is not really
relevant for that use case.

The only lacking part I found regarding frozen modules is support
for these in pkgutil.py. But that's easy to add:

--- /home/lemburg/egenix/projects/PyRun/Python-3.8.0/Lib/pkgutil.py
2019-10-14 1>
+++ ./Lib/pkgutil.py2019-11-17 11:36:38.404752218 +0100
@@ -315,20 +315,27 @@
 return self.etc[2]==imp.PKG_DIRECTORY

 def get_code(self, fullname=None):
+# eGenix PyRun needs pkgutil to also work for frozen modules,
+# since pkgutil is used by the runpy module, which is needed
+# to implement the -m command line switch.
+if self.code is not None:
+return self.code
 fullname = self._fix_name(fullname)
-if self.code is None:
-mod_type = self.etc[2]
-if mod_type==imp.PY_SOURCE:
-source = self.get_source(fullname)
-self.code = compile(source, self.filename, 'exec')
-elif mod_type==imp.PY_COMPILED:
-self._reopen()
-try:
-self.code = read_code(self.file)
-finally:
-self.file.close()
-elif mod_type==imp.PKG_DIRECTORY:
-self.code = self._get_delegate().get_code()
+mod_type = self.etc[2]
+if mod_type == imp.PY_FROZEN:
+self.code = imp.get_frozen_object(fullname)
+return self.code
+elif mod_type==imp.PY_SOURCE:
+source = self.get_source(fullname)
+self.code = compile(source, self.filename, 'exec')
+elif mod_type==imp.PY_COMPILED:
+self._reopen()
+try:
+self.code = read_code(self.file)
+finally:
+self.file.close()
+elif mod_type==imp.PKG_DIRECTORY:
+self.code = self._get_delegate().get_code()
 return self.code

 def get_source(self, fullname=None):

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Nick Coghlan]

Nick Coghlan  added the comment:

For the module metadata problem: one potential approach to that for "designed 
to be frozen" stdlib modules is to set the values directly in the module code, 
rather than trying to set them automatically in the frozen import machinery.

It should also be possible to delete the implicitly created metadata fields and 
use module level dynamic attribute retrieval to find the stdlib source code for 
tracebacks and introspection purposes without incurring any start up costs.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Tue, Aug 31, 2021 at 12:14 PM Brett Cannon  wrote:
> > set __file__ (and __path__) on frozen modules?
>
> See https://bugs.python.org/issue21736

Great!  I'll take a look.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Tue, Aug 31, 2021 at 10:05 PM Inada Naoki  wrote:
> I don't want all frozen header files to be committed in git repository.
> Can't we just build them during build process?

That's a good point (and an interesting one).  Only two of the frozen
modules are necessary (_frozen_importlib_external and
_frozen_importlib_external, to bootstrap the import system).  So for
those two it makes sense to have them in the git repository.

For all the rest it isn't necessary.  The only advantage is that
contributors don't have to think about them and they will be
guaranteed to be there.  However, if someone clones the repo they have
to build Python, so the frozen modules will get created anyway at that
point.

So I'm fine with not committing all those modules.  This will require
that all those files be added to the .gitignore file.  (I'll update my
PR accordingly.)

-eric

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Inada Naoki]

Inada Naoki  added the comment:

I don't want all frozen header files to be committed in git repository.
Can't we just build them during build process?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Change by Eric Snow :


--
keywords: +patch
pull_requests: +26549
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/28107

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Marc-Andre Lemburg]

Marc-Andre Lemburg  added the comment:

On 31.08.2021 20:14, Brett Cannon wrote:
> 
> Brett Cannon  added the comment:
> 
>> set __file__ (and __path__) on frozen modules?
> 
> See https://bugs.python.org/issue21736

The patch on that ticket is straight from PyRun, where the
__file__ location is set in a way which signals that the file
does not exist, but instead is baked into the executable:

>>> import os
>>> os.__file__
'/os.py'

Not doing this breaks too many tests in the test suite for no
good reason, which is why I mentioned "practicality beats
purity" in the ticket.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Brett Cannon]

Brett Cannon  added the comment:

> set __file__ (and __path__) on frozen modules?

See https://bugs.python.org/issue21736

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

On Mon, Aug 30, 2021 at 2:22 PM Guido van Rossum  wrote:
> I propose to only opt in by default in **PGO builds**. After all what we're 
> doing is another extreme optimization.
>
> It should always be possible to opt in using some -X flag (e.g. to debug the 
> freeze import loader) and it should also always be possible to opt *out* (for 
> those cases where you want to edit an installed stdlib module in-place in 
> anger).
>
> I don't know how the -X mechanism works exactly but probably we could make 
> the spelling
>
>   python -X freeze=on|off
>
> with a dynamic default.

+1 to all that

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

FWIW, I'd be okay with doing the -X flag in a separate PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

> At this point, here are the open questions I'm seeing:

> + The editing-stdlib-.py-files problem: [...]

> + Compatibility: [...]

? + Penalty for too many frozen modules: [...]

> FWIW, I think the ideal mechanism for a dev build will be to opt in to using 
> frozen modules (instead of the source modules).  Otherwise it is too easy for 
> the unaware contributor to waste substantial time figuring out why their 
> changes are not getting used.

Agreed. I don't care much about people (even myself) editing installed modules. 
But I care a lot about people who do a git checkout and build from source 
editing modules and testing them without doing an install (Personally I never 
install what I build from source except to test the installation process.)

> Consequently, here's my order of preference for ignoring frozen modules:

> 1. use Py_DEBUG as an opt-out flag
>(if we think contributors are editing stdlib modules on a build without 
> Py_DEBUG then that isn't good enough)
> 2. automatically skip frozen modules if it's a dev build, with an explicit 
> configure flag to opt in to frozen modules.

I propose to only opt in by default in **PGO builds**. After all what we're 
doing is another extreme optimization.

It should always be possible to opt in using some -X flag (e.g. to debug the 
freeze import loader) and it should also always be possible to opt *out* (for 
those cases where you want to edit an installed stdlib module in-place in 
anger).

I don't know how the -X mechanism works exactly but probably we could make the 
spelling

  python -X freeze=on|off

with a dynamic default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

[Gregory Szorc]


> What do you set __file__ to? [...]

Exactly. I think it should not be set, just like it's not set for builtin 
modules.

> I have some observations about the implications of this. I typed up a long 
> comment but then realized someone would probably complain about me 
> distracting from the technical parts of this issue. Which forum is most 
> appropriate for this less technical commentary? (I want to lay out the case 
> for building out an official importer far more advanced than frozen importer.)

That seems to be something for python-dev. (There used to be an "import-sig" 
but the last mention I have from it is about that list shutting down for lack 
of traffic.) Or if you're still looking for more brainstorming you could try 
python-ideas first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Guido van Rossum]

Guido van Rossum  added the comment:

> FWIW, I asked Ned Batchelder about this and he said this approach 
> ("fullcoverage" [1]) was added to support running coverage on the stdlib. 
> [...]

The docs you pointed out in [3] (where it talks about a *horrible hack you 
should never use" :-) should be amended with something explaining that "you 
need to comment out the line "" from frozen_module.py for this to 
work, else the frozen version of the encodings module will take priority over 
the imposter from "fullcoverage"."

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Gregory Szorc]

Gregory Szorc  added the comment:

> For stdlib modules it wouldn't be a big problem to set __file__ on
> frozen modules.
> Would that be enough to solve the problem?

What do you set __file__ to? Do you still materialize the .py[c] files on disk 
and set to that? (This would make the most sense.) If you support setting 
__file__ on a frozen module, how does this work at the C level? A stdlib module 
would want a different __file__ from a 3rd party module using the frozen module 
API. This would seemingly require enhancements to the frozen modules C API or 
some kind of hackery only available to stdlib frozen modules.

>> When I investigated freezing the standard library for PyOxidizer, I ran into 
>> a rash
>> of problems. The frozen importer doesn't behave like PathFinder. It doesn't
>> (didn't?) set some common module-level attributes
> This is mostly fixable for stdlib modules.  Which attributes would
> need to be added?  Are there other missing behaviors?

It was a few years ago and I can't recall specifics. I just remember that after 
encountering a couple unrelated limitations with the frozen importer I decided 
it was far too primitive to work as a general importer and decided I'd have to 
roll my own.

> Good point.  Supporting more of the FileLoader API on the frozen
> loader is something to look into, at least for stdlib modules.

>> I agree that we should shore up the frozen importer -- probably in a 
>> separate PR though.
>> (@Eric: do you think this is worth its own bpo issue?)
> Yeah.

I have some observations about the implications of this. I typed up a long 
comment but then realized someone would probably complain about me distracting 
from the technical parts of this issue. Which forum is most appropriate for 
this less technical commentary? (I want to lay out the case for building out an 
official importer far more advanced than frozen importer.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

> * tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) 
> may lose their entry point

FWIW, I asked Ned Batchelder about this and he said this approach 
("fullcoverage" [1]) was added to support running coverage on the stdlib.  It 
doesn't affect other users of coverage.py.  He didn't have any info on where 
this is used currently, though I'm pretty sure we do run coverage in CI.  
Furthermore, the devguide talks about running coverage.py on the stdlib and 
initially indicates that modules imported during startup are not covered. [2]  
However, it does have a section talking about using "fullcoverage" to cover 
startup modules.

I expect that any solution we make for contributors editing stdlib source files 
will resolve the potential issue for coverage.py.


[1] https://github.com/nedbat/coveragepy/tree/master/coverage/fullcoverage
[2] https://devguide.python.org/coverage/?highlight=coverage#measuring-coverage
[3] 
https://devguide.python.org/coverage/?highlight=coverage#coverage-results-for-modules-imported-early-on

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45020] Freeze all modules imported during startup.

[Eric Snow]

Eric Snow  added the comment:

At this point, here are the open questions I'm seeing:

+ The editing-stdlib-.py-files problem:
   * use a command-line flag to opt-out of frozen modules?
   * use a build flag to opt out (e.g. a configure flag or a new Py_NO_FROZEN 
or even Py_DEBUG)?
   * ignore frozen modules if it's a dev build?
   * (note: importlib._bootstrap and importlib._bootstrap_external must always 
be frozen, regardless of a flag)
   * accommodate users of an installed Python that sometimes edit stdlib 
modules while debugging code?
   * always emit a warning if a frozen module is ignored (in favor of the 
source module)?

+ Compatibility:
   * set __file__ (and __path__) on frozen modules?
   * store the hash of the source file for frozen modules (in the frozen .h 
files)?
   * support more of the FileLoader API on the frozen loader?

+ Penalty for too many frozen modules:
   * should we only freeze the UTF-8 encoding?
   * should we use something other than linear search for looking up frozen 
modules?

FWIW, I think the ideal mechanism for a dev build will be to opt in to using 
frozen modules (instead of the source modules).  Otherwise it is too easy for 
the unaware contributor to waste substantial time figuring out why their 
changes are not getting used.

Consequently, here's my order of preference for ignoring frozen modules:

1. use Py_DEBUG as an opt-out flag
   (if we think contributors are editing stdlib modules on a build without 
Py_DEBUG then that isn't good enough)
2. automatically skip frozen modules if it's a dev build, with an explicit 
configure flag to opt in to frozen modules

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com