[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Is the issue done? Can it be closed?

--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Brett Cannon]

Brett Cannon added the comment:

To be honest, I'm not excited by ./configure --enable-optimizations. I was 
happy with an explicit "make profile-opt".

The configure flag was added because not everything can be enabled just in the 
Makefile and so that people have a single flag to set to turn on any and all 
optimizations that work consistently across OSs (it used to be LTO+PGO until we 
discovered how universally broken LTO seems to be).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Tomas Orsava added the comment:

> Oh wait, I expected that "make test" would build Python with PGO, but it
> doesn't anymore. Would it be possible to change the Makefile to make sure
> that "make python" (on Linux, "make python.exe" on macOS) would build Python
> using PGO?
>
> I tried to hack something, but I'm lost in dependencies... "make profile-opt"
> runs "$(MAKE) build_all_generate_profile" which runs "$(MAKE) build_all
> CFLAGS_NODIST=..." and build_all depends on "$(BUILDPYTHON)".
>
> I'm not sure that it's possible to make everything automatic because of the
> high number of targets and dependencies.

In addition, profile-opt would have to rewritten to test whether there's
already an interpreter built with PGO. And if there is, regenerate it only if
the main target is `all` or `profile-opt`, but not when someone runs `make
test`, `make install`, etc.

It's maybe too radical, but we could change it so that if you run `make test`, 
`make install` and the like and there's no interpreter built already to test or 
install, the user is told to run `make` first.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:

Oh wait, I expected that "make test" would build Python with PGO, but it 
doesn't anymore. Would it be possible to change the Makefile to make sure that 
"make python" (on Linux, "make python.exe" on macOS) would build Python using 
PGO?

I tried to hack something, but I'm lost in dependencies... "make profile-opt" 
runs "$(MAKE) build_all_generate_profile" which runs "$(MAKE) build_all 
CFLAGS_NODIST=..." and build_all depends on "$(BUILDPYTHON)".

I'm not sure that it's possible to make everything automatic because of the 
high number of targets and dependencies.

To be honest, I'm not excited by ./configure --enable-optimizations. I was 
happy with an explicit "make profile-opt".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Gregory P. Smith]

Gregory P. Smith added the comment:

Yes, this appears true.

I'm okay with it because I've always considered the make, test, and install as 
different command invocations.  The package building automation I am aware of 
tends to do the same.  Our buildbots keep the steps separate.

But we need document this requirement somewhere.

I think if we wanted test and install to accurately figure out if they need to 
redo the profile build and profiling run, we'd need make to detect that _any_ 
of its possible profile build+run input files are more recent than the last 
profiling run.  ugh.

--
stage:  -> commit review
versions: +Python 2.7, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Xiang Zhang]

Xiang Zhang added the comment:

> I don't think that the patch changes the behaviour of "make": it only
> changes the behaviour of "make install" and "make test" (and a few
> others).

I don't mean "make" is changed. I mean the latter. Before, a single "make test" 
or "make install" will do all things for you, including the profile procedure. 
Now they seems not. You have to run "make" before them to trigger the profile 
procedure.

Actually I am quite okay with this change. But don't know others.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:

Xiang Zhang added the comment:
> So now we have to run make if we want to trigger the profile procedure since 
> other make commands like make test won't trigger the procedure by itself, 
> right?

I don't think that the patch changes the behaviour of "make": it only
changes the behaviour of "make install" and "make test" (and a few
others).

To be honest, I don't know if Makefile knows when a rebuild is needed
and how it works. To run benchmarks, I always rebuild Python from
scratch, even from an empty directory!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Xiang Zhang]

Xiang Zhang added the comment:

So now we have to run make if we want to trigger the profile procedure since 
other make commands like make test won't trigger the procedure by itself, right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:


New changeset a473a73d0cb42c534a3047bbf781b3c592fc33ca by Victor Stinner 
(torsava) in branch '2.7':
[2.7] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) 
(#1522)
https://github.com/python/cpython/commit/a473a73d0cb42c534a3047bbf781b3c592fc33ca


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Changes by Tomas Orsava :


--
pull_requests: +1621

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:


New changeset 8489409bbfabb2ddc30ed55c9f4d679a3710ebe4 by Victor Stinner 
(torsava) in branch '3.5':
[3.5] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) 
(#1520)
https://github.com/python/cpython/commit/8489409bbfabb2ddc30ed55c9f4d679a3710ebe4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Changes by Tomas Orsava :


--
pull_requests: +1620

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:


New changeset 03b8a378dfa46372b96790f82c85e9b72518f1bf by Victor Stinner 
(torsava) in branch '3.6':
[3.6] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) 
(#1518)
https://github.com/python/cpython/commit/03b8a378dfa46372b96790f82c85e9b72518f1bf


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Changes by Tomas Orsava :


--
pull_requests: +1618

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:

I closed my issue #29641 as a duplicate of this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[STINNER Victor]

STINNER Victor added the comment:


New changeset a1054c3b0037d4c2a5492e79fc193f36245366c7 by Victor Stinner 
(torsava) in branch 'master':
bpo-29243: Fix Makefile with respect to --enable-optimizations (#1478)
https://github.com/python/cpython/commit/a1054c3b0037d4c2a5492e79fc193f36245366c7


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Tomas Orsava added the comment:

I've opened a PR#1478 that I believe fixes the issue.

--
nosy: +torsava

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Tomas Orsava]

Changes by Tomas Orsava :


--
pull_requests: +1577

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Charalampos Stratakis]

Changes by Charalampos Stratakis :


--
nosy: +cstratak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Xiang Zhang]

Xiang Zhang added the comment:

Another complaint from #29889.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[INADA Naoki]

INADA Naoki added the comment:

That's why I want to enable only --with-lto

--
nosy: +inada.naoki

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Brett Cannon]

Brett Cannon added the comment:

Since optimizations uses PGO which should do a fresh run, that's why there's 
the clearing of the state.

I'm personally not bothered by it as you really should only being doing an 
optimized build once when you do the first install and then you're done. But if 
or anyone else can come up with a way to not clear out the results while not 
accidentally using stale PGO-collected stats or builds then I don't see why we 
can't speed it up.

--
nosy: +brett.cannon
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29243] --enable-optimizations makes common build commands always need to compile from scratch

[Xiang Zhang]

New submission from Xiang Zhang:

In 3.6 we get --enable-optimizations. One thing I find annoyed with this flag 
is that it makes some build commands compile from scratch since it always 
clears the environment first. For example, with the commands listed at the top 
of Makefile:

./configure
make 
make test
make install

It compiles 3 times and considering the optimized build needs more time every 
time, it lasts long.

I am not sure this is a problem and feel free to close it.

--
components: Build
messages: 285233
nosy: xiang.zhang
priority: normal
severity: normal
status: open
title: --enable-optimizations makes common build commands always need to 
compile from scratch
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com