[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 2021-09-28 at 10:22 +0200, Marc-Andre Lemburg wrote: > On 27.09.2021 18:51, Eric Snow wrote: > > We've frozen most of the stdlib modules imported during "python -c > > pass" [1][2], to make startup a bit faster.  Import of those modules > > is controlled by "-X frozen_modules=[on|off]". 

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sept 2021 at 15:33, Eric Snow wrote: > > It means that the site module module can no longer be "customized" by > > modifying directly the site.py file (inject a path in PYTHONPATH env > > var where the customized site.py lives). But there is already a > > supported way to customize

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:55 AM Eric V. Smith wrote: > As a compromise, how about go with #1, but print a warning if python > detects that it's not built with optimizations or is run from a source > tree (the conditions in #2 and #3)? The warning could suggest running > with "-X

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:47 AM Pablo Galindo Salgado wrote: > One interesting consequence of what Eric mentioned (They have a different > loader and repr. Also, frozen modules do not > have __file__ set (and __path__ is always []).) is that frozen modules don't > have a `__file__` attribute

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:36 AM Victor Stinner wrote: > Honestly, for me, #1: always on, is the most reasonable choice. > > I dislike when Python behaves differently depending on subtle things > like "was it built with optimizations" or "is Python started from its > source tree"? > > When I built

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 2:22 AM Marc-Andre Lemburg wrote: > #3 sounds like a good solution, but how would you detect "running > from the source tree" ? This sounds like you need another stat call > somewhere, which is what the frozen modules try to avoid. We already look for the stdlib dir in

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 6:02 AM Ronald Oussoren via Python-Dev wrote: > Of course. I mentioned it because the proposal is to add a new option that’s > enabled after installation, and basically not when the testsuite is run. > That’s not a problem, we could just enable the option in most CI

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 2:54 AM Ronald Oussoren via Python-Dev wrote: > I agree, but… Most CPython tests are run while running from the source tree, > that means that there will have to be testrunner configurations that run with > “-X frozen_modules=on”. If the build option that determines the

[Python-Dev] Re: The Default for python -X frozen_modules.

The Python Debug Build document lists changes compared to a release build: https://docs.python.org/dev/using/configure.html#python-debug-build Sometimes, I'm confused that "./python" (Python built locally in debug mode) displays warnings, whereas "python" (Fedora package) doesn't. See also the

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/28/2021 9:17 AM, Antoine Pitrou wrote: On Tue, 28 Sep 2021 09:14:38 -0400 "Eric V. Smith" wrote: On 9/28/2021 9:10 AM, Antoine Pitrou wrote: On Tue, 28 Sep 2021 08:55:05 -0400 "Eric V. Smith" wrote: So I prefer to teach everybody how to use "-X frozen_modules=off" if they want to hack

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 09:14:38 -0400 "Eric V. Smith" wrote: > On 9/28/2021 9:10 AM, Antoine Pitrou wrote: > > On Tue, 28 Sep 2021 08:55:05 -0400 > > "Eric V. Smith" wrote: > >>> So I prefer to teach everybody how to use "-X frozen_modules=off" if > >>> they want to hack the stdlib for their

[Python-Dev] Re: The Default for python -X frozen_modules.

On 28.09.2021 14:26, Filipe Laíns wrote: > On Tue, 2021-09-28 at 10:22 +0200, Marc-Andre Lemburg wrote: >> On 27.09.2021 18:51, Eric Snow wrote: >>> We've frozen most of the stdlib modules imported during "python -c >>> pass" [1][2], to make startup a bit faster.  Import of those modules >>> is

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/28/2021 9:10 AM, Antoine Pitrou wrote: On Tue, 28 Sep 2021 08:55:05 -0400 "Eric V. Smith" wrote: So I prefer to teach everybody how to use "-X frozen_modules=off" if they want to hack the stdlib for their greatest pleasure. I prefer that such special use case requires an opt-in option,

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 08:55:05 -0400 "Eric V. Smith" wrote: > > > > So I prefer to teach everybody how to use "-X frozen_modules=off" if > > they want to hack the stdlib for their greatest pleasure. I prefer > > that such special use case requires an opt-in option, the special use > > case is not

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/28/2021 8:36 AM, Victor Stinner wrote: On Mon, Sep 27, 2021 at 6:58 PM Eric Snow wrote: We've frozen most of the stdlib modules imported during "python -c pass" [1][2], to make startup a bit faster. Import of those modules is controlled by "-X frozen_modules=[on|off]". Currently it

[Python-Dev] Re: The Default for python -X frozen_modules.

Would it be possible to add a __file__ attribute? Victor On Tue, Sep 28, 2021 at 2:47 PM Pablo Galindo Salgado wrote: > > > What is the annoyance? What is different between frozen and not frozen? > > One interesting consequence of what Eric mentioned (They have a different > loader and repr.

[Python-Dev] Re: The Default for python -X frozen_modules.

> What is the annoyance? What is different between frozen and not frozen? One interesting consequence of what Eric mentioned (They have a different loader and repr. Also, frozen modules do not have __file__ set (and __path__ is always []).) is that frozen modules don't have a `__file__`

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to default

[Python-Dev] Re: The Default for python -X frozen_modules.

> On 28 Sep 2021, at 10:54, Antoine Pitrou wrote: > > On Tue, 28 Sep 2021 10:51:53 +0200 > Ronald Oussoren via Python-Dev wrote: >>> On 28 Sep 2021, at 10:05, Antoine Pitrou wrote: >>> >>> On Mon, 27 Sep 2021 10:51:43 -0600 >>> Eric Snow >> > wrote:

[Python-Dev] Re: The Default for python -X frozen_modules.

> On 28 Sep 2021, at 10:05, Antoine Pitrou wrote: > > On Mon, 27 Sep 2021 10:51:43 -0600 > Eric Snow mailto:ericsnowcurren...@gmail.com>> > wrote: >> We've frozen most of the stdlib modules imported during "python -c >> pass" [1][2], to make startup a bit faster. Import of those modules >>

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, 28 Sep 2021 10:51:53 +0200 Ronald Oussoren via Python-Dev wrote: > > On 28 Sep 2021, at 10:05, Antoine Pitrou wrote: > > > > On Mon, 27 Sep 2021 10:51:43 -0600 > > Eric Snow > > wrote: > >> We've frozen most of the stdlib modules imported during

[Python-Dev] Re: The Default for python -X frozen_modules.

On 27.09.2021 18:51, Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to default to "on".

[Python-Dev] Re: The Default for python -X frozen_modules.

On 28.09.2021 10:22, Marc-Andre Lemburg wrote: > On 27.09.2021 18:51, Eric Snow wrote: >> We've frozen most of the stdlib modules imported during "python -c >> pass" [1][2], to make startup a bit faster. Import of those modules >> is controlled by "-X frozen_modules=[on|off]". Currently it

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, 27 Sep 2021 10:51:43 -0600 Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 12:52 AM Eric Snow wrote: > On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner wrote: > > Which stdlib modules are currently frozen? If I really want to hack > > site.py or os.py for whatever reason, I just have to use "python3 -X > > frozen_modules=off"? > > The

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner wrote: > Which stdlib modules are currently frozen? If I really want to hack > site.py or os.py for whatever reason, I just have to use "python3 -X > frozen_modules=off"? The single-source-of-truth is Tools/scripts/freeze_modules.py. After running

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 3:04 PM Barry Warsaw wrote: > If you’re planning a runtime -X option, then does that mean that the modules > will be frozen at build time but Python will decide at runtime whether to use > the frozen modules or the unfrozen ones? Correct. FYI, this was already done. >

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 2:59 PM Brett Cannon wrote: > What about opting out when `--with-pydebug` is used? I'm not sure how many > people actively develop in a non-debug build other than testing something, > but at that point I would be having to run `make` probably anyway for > whatever I'm

[Python-Dev] Re: The Default for python -X frozen_modules.

Hi Eric, Which stdlib modules are currently frozen? If I really want to hack site.py or os.py for whatever reason, I just have to use "python3 -X frozen_modules=off"? > 1. always default to "on" (the annoyance for contributors isn't big enough?) What is the annoyance? What is different between

[Python-Dev] Re: The Default for python -X frozen_modules.

A couple of questions. If you’re planning a runtime -X option, then does that mean that the modules will be frozen at build time but Python will decide at runtime whether to use the frozen modules or the unfrozen ones? Are you planning on including the currently frozen importlib modules in

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 9:54 AM Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to default

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 12:40 PM Steve Dower wrote: > Having it be implied by an "--enable-optimizations" option is totally > fine (and we'd add one to build.bat for this), but I still think it > needs to be discoverable later whether the frozen modules build option > was used or not, independent

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/27/2021 7:15 PM, Ethan Furman wrote: On 9/27/21 10:50 AM, Guido van Rossum wrote: > So my *actual* proposal (call it #2') is to use a separate compile-time flag, which is set by `./configure > --enable-optimizations` regardless of whether PGO/LTO are possible, and which on Windows can

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/27/2021 6:40 PM, Patrick Reader wrote: I accidentally a word. "... module's hash and/or *timestamp* and comparing it ..." On 27/09/2021 18:38, Patrick Reader wrote: How about checking each non-frozen module's hash and/or and comparing it to that of the frozen module? Would that defeat

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/27/21 10:50 AM, Guido van Rossum wrote: > So my *actual* proposal (call it #2') is to use a separate compile-time flag, which is set by `./configure > --enable-optimizations` regardless of whether PGO/LTO are possible, and which on Windows can be set by > `PCbuild\build.bat --pgo` (we

[Python-Dev] Re: The Default for python -X frozen_modules.

I accidentally a word. "... module's hash and/or *timestamp* and comparing it ..." On 27/09/2021 18:38, Patrick Reader wrote: > How about checking each non-frozen module's hash and/or and comparing it to > that of the frozen module? Would that defeat the performance improvement of > freezing?

[Python-Dev] Re: The Default for python -X frozen_modules.

How about checking each non-frozen module's hash and/or and comparing it to that of the frozen module? Would that defeat the performance improvement of freezing? Is it just a terrible idea? On 27/09/2021 17:51, Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 10:40 AM Steve Dower wrote: > On 9/27/2021 5:51 PM, Eric Snow wrote: > > Possible solutions: > > > > 1. always default to "on" (the annoyance for contributors isn't big > enough?) > > 2. default to "on" if it's a PGO build (and "off" otherwise) > > Just to air my concerns

[Python-Dev] Re: The Default for python -X frozen_modules.

On 9/27/2021 5:51 PM, Eric Snow wrote: Possible solutions: 1. always default to "on" (the annoyance for contributors isn't big enough?) 2. default to "on" if it's a PGO build (and "off" otherwise) Just to air my concerns regarding option 2 (which I've already spoken to Eric about): I feel

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 3:21 AM Eric Snow wrote: > > On Mon, Sep 27, 2021 at 11:09 AM Chris Angelico wrote: > > When exactly does the freezing happen? > > When you build the executable (e.g. "make -j8", > ".\PCbuild\build.bat"). So your changes to those .py files wouldn't > show up until then.

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 10:51 AM Eric Snow wrote: > Possible solutions: > > 1. always default to "on" (the annoyance for contributors isn't big enough?) > 2. default to "on" if it's a PGO build (and "off" otherwise) > 3. default to "on" unless running from the source tree FWIW, I'm planning on

[Python-Dev] Re: The Default for python -X frozen_modules.

On Mon, Sep 27, 2021 at 11:09 AM Chris Angelico wrote: > When exactly does the freezing happen? When you build the executable (e.g. "make -j8", ".\PCbuild\build.bat"). So your changes to those .py files wouldn't show up until then. -eric ___

[Python-Dev] Re: The Default for python -X frozen_modules.

On Tue, Sep 28, 2021 at 2:58 AM Eric Snow wrote: > > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to