[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[STINNER Victor]

STINNER Victor  added the comment:

On my PR, Nick Coghlan wrote:

"We may want to tweak the porting note in What's New, as I believe this may 
read more environment variables at init time than the interpreter did 
previously, so embedding applications that want more complete control may now 
need to set Py_IgnoreEnvironmentFlag when they could previously get by without 
it."

https://github.com/python/cpython/pull/8659#pullrequestreview-143397378

I'm not sure about this, since it's really a bug of Python 3.7.0 and it's now 
fixed.

I let you decide if the issue should be closed or not, I leave it open.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 0c90d6f75931da4fec84d06c2efe9dd94bb96b77 by Victor Stinner in 
branch '3.7':
[3.7] bpo-34247: Fix Python 3.7 initialization (#8659)
https://github.com/python/cpython/commit/0c90d6f75931da4fec84d06c2efe9dd94bb96b77


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Ronald Oussoren  added the comment:

The PR works for me as well, and looks ok to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[STINNER Victor]

STINNER Victor  added the comment:

I tested manually attached t.c with PR 8659: I get optimize=2 as expected. Full 
output:

sys.flags(debug=0, inspect=0, interactive=0, optimize=2, dont_write_bytecode=0, 
no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, 
quiet=0, hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[STINNER Victor]

STINNER Victor  added the comment:

I first proposed to backport all code from master:
https://mail.python.org/pipermail/python-dev/2018-July/154882.html

But I'm not sure that it's a good idea, since I made *many* changes in the 
master branch since 3.7... I even added more files, and the _PyCoreConfig API 
is going to change anyway...

So I wrote a simpler PR 8659 which backports unit tests, adapt them for Python 
3.7, and fix this issue: PYTHONOPTIMIZE and other environment variables and now 
read by Py_Initialize(), not only by Py_Main().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +8153

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

I believe Victor is at the EuroPython sprints as well, so if I'm right about 
that, I think the best option would be for the two of you to work out a 
resolution for 3.7.1 in person that at least keeps the test suites reasonably 
consistent, even if the implementations have diverged (the new tests already 
added for bpo-34170 were the main reason I stopped working on my initial patch 
for this).

And agreed on fixing the current dependency inversion between pylifecycle and 
the py_main code - the functions needed by both Py_Main and Py_Initialize 
should be migrated down the stack into pylifecycle as part of the bpo-34170 
work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I've created a pull request that seems to work properly, although I am not 
entirely sure that this is the right way to fix this.

As I mentioned in the pull request I created it against the 3.7 branch because 
of bpo-34170, but would happily rebase it for master.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Change by Ronald Oussoren :


--
keywords: +patch
pull_requests: +8037
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Ronald Oussoren  added the comment:

Interesting... pylifecycle.c uses code in main.c, I hadn't expected that.

If I've read the code correctly Py_Initialize won't look at PYTHONOPTIMZE at 
all (and other environment variables that have a command-line equivalent). 
Those are read in main.c:cmdline_get_env_flags, which is only called 
(indirectly) from Py_Main and _Py_UnixMain. 

I'm note sure what's the correct solution for this.

1. Reading PYTHONOPTIMZE (and related variables) could be done 
   in _PyCoreConfig_Read, but that would then need to grow a flag to ensure
   that this won't replace values calculated earlier in the implementation 
   of Py_Main

2. Writing back the global variables is probably best done using a new
   function (lets _PyCoreConfig_SetGlobals), mirroring
   pymain_set_global_config but using a _PyCoreConfig config structure
   instead of the command-line structure used by the latter function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Ronald Oussoren  added the comment:

FYI: I've filed issue34255 while working on this, test_embed assumes that 
you're building in the source directory (which I usually don't do).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Ronald Oussoren  added the comment:

I'm at the EuroPython sprints and currently working on a testcase for this. 
Should be a nice way to get back into actively contributing to CPython :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

This seems closely related to the work Victor is already pursuing to resolve 
bpo-34170 for Python 3.8 (e.g. 
https://github.com/python/cpython/commit/56b29b6d6fa3eb32bb1533ee3f21b1e7135648a0
 ).

The bpo-34170 changes are more invasive than we'd like for a maintenance 
release, but it would be preferable to keep at least any new Python 3.7 test 
cases somewhat aligned with their Python 3.8 counterparts.

--
assignee:  -> vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

Next steps:

- add a test case that runs the same sys.flags checks as in 
https://github.com/python/cpython/commit/d7ac06126db86f76ba92cbca4cb702852a321f78,
 but as a test_embed embedding test using Py_Initialize rather than Py_Main
- create a shared internal API that both Py_Main and Py_Initialize can use to 
write the environment variable settings back to the global variables

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

Ned, I think we should consider this a release blocker for 3.7.1

--
nosy: +ned.deily
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

Comparing to Python 3.6, we can see that the code to update the C global flags 
from the environment variables used to live directly in 
_PyInitializeEx_Private: 
https://github.com/python/cpython/blob/9d85856044a2c7d681ea38b5ff99af375b228a0f/Python/pylifecycle.c#L305

In Python 3.7 that section 
(https://github.com/python/cpython/blob/3.7/Python/pylifecycle.c#L913 ) 
currently calls _PyCoreConfig_Read instead, which *only* updates the config 
struct - it doesn't update the C level global variables.

By contrast, Py_Main eventually calls 
https://github.com/python/cpython/blob/7cbde0e09daba4259565738e48f141851287fe29/Modules/main.c#L1394
 which handles writing the state from the config struct back to the global 
variables.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

Ah, wait - I see a relevant difference: the test case for issue31845 uses 
Py_Main, *not* Py_Initialize (since it's a command line test, not an embedding 
test).

That means it's currently possible for the sequence of operations in 
https://github.com/python/cpython/blob/7cbde0e09daba4259565738e48f141851287fe29/Python/pylifecycle.c#L913
 to be wrong (or missing steps) without causing the test suite to fail.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Nick Coghlan]

Nick Coghlan  added the comment:

This feels like a duplicate of https://bugs.python.org/issue31845 (see 
https://github.com/python/cpython/commit/d7ac06126db86f76ba92cbca4cb702852a321f78
 )

Is this definitely with the 3.7.0 release, or is it potentially with one of the 
earlier alphas before that fix was applied?

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Change by Ronald Oussoren :


Added file: https://bugs.python.org/file47717/foo.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

Change by Ronald Oussoren :


Added file: https://bugs.python.org/file47716/t.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34247] PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher

[Ronald Oussoren]

New submission from Ronald Oussoren :

Attached is a very basic launcher for python scripts. This script works as 
expected in Python 3.6, but not in 3.7.0.

In particular, the launcher sets the environment variable PYTHONOPTIMIZE before 
calling Py_Initialize and that setting is ignored.

I haven't tried testing with the tip of the tree yet.

--
components: Interpreter Core
keywords: 3.7regression
messages: 322487
nosy: ronaldoussoren
priority: normal
severity: normal
status: open
title: PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com