[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Recently, some people asked me for an update for my FAT Python project. So I 
rebased this change I wrote 1 year 1/2 and adapted it for the new code base:

* I renamed test_pep511.py to test_code_transformer.py
* I removed the sys module from the PyInterpreterState structure (added in my 
previous patch), since Eric Snow and Nick Coghlan removed a variant of the sys 
module from this structure for their work on subinterpreters and reworked 
Python initialization

The PEP 511 is not accepted, so the implementation is still a work-in-progress 
(WIP) and must not be merged.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2401

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

TODO: implement a fast-path for PyCode_Optimize() avoiding completly the 
"expensive" code transformer API, before the first call to 
sys.set_code_transformer().

It would avoid list <=> tuple conversions for code constants, and the need of 
creating two code objects.

In practice, this fast-path should be taken in most cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Patch version 6: updated to the new code_transformer() API, it now takes a code 
object as input and must return a new code object.

Sadly, it looks like there are reference leaks (try: ./python -m test -R 3:3 
test_import).

--
Added file: http://bugs.python.org/file41912/transformers-6.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Rebased patch combining also pycf_transformed_ast.patch.

--
Added file: http://bugs.python.org/file41816/transformers-5.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

> Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

Additionnal pycf_transformed_ast.patch implements it. The patch should be 
applied on top of transformers-4.patch.

Note: PyCF_TRANSFORMED_AST has the same value (0x1000) than the old constant 
CO_GENERATOR_ALLOWED. This constant was removed in Python 2.5.0 by the 
changeset 6b42920accc9 in 2006. Is it an issue?

--
Added file: http://bugs.python.org/file41700/pycf_transformed_ast.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Patch version 4: Fix sys_set_code_transformers(), initialize seq to NULL to fix 
a crash on error handling.

--
Added file: http://bugs.python.org/file41695/transformers-4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Rebased patch.

--
Added file: http://bugs.python.org/file41692/transformers-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Note: ast.PyCF_TRANSFORMED_AST is not implemented yet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

(oops, i posted the same message with patch twice)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Patch version 2:

* allow "-" character in the argument of the -o command line option
* add an unit test on invalid type for code transformer
* fix a crash in sys.set_code_transformers(): replace Py_DECREF() with 
Py_XDECREF()

--
Added file: http://bugs.python.org/file41656/transformers-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

Changes by STINNER Victor :


--
Removed message: http://bugs.python.org/msg258579

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

Changes by STINNER Victor :


Removed file: http://bugs.python.org/file41655/transformers-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Patch version 2:

* allow "-" character in the argument of the -o command line option
* add an unit test on invalid type for code transformer
* fix a crash in sys.set_code_transformers(): replace Py_DECREF() with 
Py_XDECREF()

--
Added file: http://bugs.python.org/file41655/transformers-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

sys_init_code_transformer: Py_DECREF(name); must be Py_XDECREF(name); at exit, 
it can NULL. An unit test on an invalid transformer (with no name) is missing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

main.c:


+case 'o':
+if (wcslen(_PyOS_optarg) == 0
+   || wcschr(_PyOS_optarg, L'.')
+   || wcschr(_PyOS_optarg, L'-')
...

'-' character must be valid for -o argument.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

STINNER Victor added the comment:

Missing patch for test_sys testing the new sys.get/set_code_transformers() 
functions.

--
Added file: http://bugs.python.org/file41644/test_sys.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26145] PEP 511: Add sys.set_code_transformers()

[STINNER Victor]

New submission from STINNER Victor:

Attached patch implements the largest part of the PEP 511:

* add sys.get_code_transformers() and sys.set_code_transformers()
* add sys.implementation.optim_tag
* add -o command line option setting sys.implementation.optim_tag
* modify importlib to use sys.implementation.optim_tag to build .pyc filenames
* modify importlib to raise an ImportError when the .pyc file is missing, the 
.py is available, and the optimizer tag created from code transformers is 
different than the current sys.implementation.optim_tag
* add Lib/test/test_pep511.py: test changes especially bytecode and AST 
transformers
* add a PeepholeOptimizer object created by default for 
sys.set_code_transformers()
* skip the peephole optimizer when -o noopt is used
* add sys field to the PyInterpreterState structure: used by 
_PySys_GetCodeTransformers() to get code transformers
* update unit tests to use the optimizer to create the .pyc filename

An optimizer tag must contain characters: '.', '-', directory separators or NUL 
character.

Brett Canon proposed to not include the optimizer tag in .pyc filenames if it 
is "opt" to limit backward incompatible changes.

Note: The PEP 511 is under discussion on python-ideas. The patch implements the 
first version of the PEP, sent to python-ideas, so it doesn't include proposed 
changes like changing code_transformer() prototype to "def 
code_transformer(self, code, context)".

See also the issue #26100: add test.support.optim_args_from_interpreter_flags().

--
files: transformers.patch
keywords: patch
messages: 258509
nosy: brett.cannon, haypo
priority: normal
severity: normal
status: open
title: PEP 511: Add sys.set_code_transformers()
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file41643/transformers.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com