[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Petr Viktorin 

On 23. 11. 21 18:00, Victor Stinner wrote:

I completed the PEP: https://python.github.io/peps/pep-0670/



What I don't like about this PEP is that it documents changes that were 
already pushed, not planned ones. But, what's done is done...
Are there more macros that are yet to be converted to macros, other than 
the ones in GH-29728? If so, can you give a list?


Since this is about converting existing macros (and not writing new 
ones), can you talk about which of the "macro pitfalls" apply to the 
macros in CPython that were/will be changed?


The "Backwards Compatibility" section is very small. Can you give a list 
of macros which lost/will lose "return values"?
Can you add the fact that some macros now can't be used as l-values? 
(and list which ones?) This change is also breaking existing code.
Are there any other issues that break existing code? (Even code that, 
for example, shouldn't work according to Python documentation, but still 
works fine in practice.)



The "Cast to PyObject*" section talks about adding new private functions 
like _Py_TYPE, which are type-safe, but keeping old names (like Py_TYPE) 
as macros that do a cast.
Could the newly added functions be made public from the start? (They 
could use names like Py_Type.) This would allow languages that don't 
have macros to use them directly, and if the non-typesafe macros are 
ever discouraged/deprecated/removed, this would allow writing compatible 
code now.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/TBIGFWSBOGW55DBRUYURR5QKRVIK3B6I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Victor Stinner 
On Wed, Nov 24, 2021 at 10:59 AM Petr Viktorin  wrote:
> Are there more macros that are yet to be converted to macros,

I suppose that you mean "to be converted to functions". Yes, there are
many, it's the purpose of the PEP.

I didn't provide a list. I would prefer to do it on a case by case
basis, as I did previously.

To answer your question: it's basically all macros, especially the
ones defined by the public C API, except the ones excluded by the PEP:
https://www.python.org/dev/peps/pep-0670/#convert-macros-to-static-inline-functions


> other than the ones in GH-29728?

The purpose of this PR is only to run benchmarks to compare the
performance of macros versus static inline functions. The PR title is
"Convert static inline to macros": it converts existing Python 3.11
static inline functions back to Python 3.6/3.7 macros. It's basically
the opposite of the PEP ;-)


> The "Backwards Compatibility" section is very small. Can you give a list
> of macros which lost/will lose "return values"?

https://bugs.python.org/issue45476 lists many of them. See also:
https://github.com/python/cpython/pull/28976


> Can you add the fact that some macros now can't be used as l-values?

If you are are talking about my merged change preventing using
Py_TYPE() as an l-value, this is out of the scope of the PEP on
purpose.

Py_TYPE(), Py_REFCNT() and Py_SIZE() could be used an l-value in
Python 3.9, but it's no longer the case in Python 3.11. Apart of that,
I'm not aware of other macros which could be "abused" as l-value.

There are macros which can be "abused" ("used") to access to structure
members and object internals. For example, &PyTuple_GET_ITEM(tuple, 0)
and &PyList_GET_ITEM(list, 0) can be "abused" to access directly to an
array of PyObject* (PyObject** type) and so modify directly a
tuple/list. I would like to change that (disallow it), but it's out of
the scope of the PEP. See https://bugs.python.org/issue41078 for my
previous failed attempt (it broke too many things). But this is more
in the scope of the PEP 620 which is a different PEP.


> Are there any other issues that break existing code?

I listed all known backward incompatibles changes in the Backward
Compatibility section. I'm not aware of other backward incompatible
changes caused by the PEP.

Converting macros to static inline functions or regular functions
didn't change the API for the macros already converted, the ones
listed in the PEP.


> The "Cast to PyObject*" section talks about adding new private functions
> like _Py_TYPE, which are type-safe, but keeping old names (like Py_TYPE)
> as macros that do a cast.
> Could the newly added functions be made public from the start? (They
> could use names like Py_Type.) This would allow languages that don't
> have macros to use them directly, and if the non-typesafe macros are
> ever discouraged/deprecated/removed, this would allow writing compatible
> code now.

I don't want to increase the size of the C API and so I chose to make
the inner function accepting PyObject* private.

I see the addition of an hypothetical Py_Type() function as an
increase of the maintenance burden: we would have to maintain it,
document it, maybe add it to the limited C API / stable ABI, write
tests, etc.

I prefer to restrict the scope of the PEP. If you want to add variants
only accepting PyObject*, that's fine, but I suggest to open a
separated issue / PEP. Also, it can be discussed on a case by case
basic (function per function).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/LYDQ2TDTPYTDIIFLTUMNPOITCOTHZOKA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Petr Viktorin 

 On 24. 11. 21 13:20, Victor Stinner wrote:

On Wed, Nov 24, 2021 at 10:59 AM Petr Viktorin  wrote:

Are there more macros that are yet to be converted to macros,


I suppose that you mean "to be converted to functions". Yes, there are
many, it's the purpose of the PEP.

I didn't provide a list. I would prefer to do it on a case by case
basis, as I did previously.

To answer your question: it's basically all macros, especially the
ones defined by the public C API, except the ones excluded by the PEP:
https://www.python.org/dev/peps/pep-0670/#convert-macros-to-static-inline-functions



other than the ones in GH-29728?


The purpose of this PR is only to run benchmarks to compare the
performance of macros versus static inline functions. The PR title is
"Convert static inline to macros": it converts existing Python 3.11
static inline functions back to Python 3.6/3.7 macros. It's basically
the opposite of the PEP ;-)



The "Backwards Compatibility" section is very small. Can you give a list
of macros which lost/will lose "return values"?


https://bugs.python.org/issue45476 lists many of them. See also:
https://github.com/python/cpython/pull/28976


Can you put this in the PEP? If things should be evaluated on a 
case-by-case basiswe should know about the cases.


Also, this PR is about preventing the use of some macros as l-values, 
which you say is out of scope for the PEP. I'm connfused.



Can you add the fact that some macros now can't be used as l-values?


If you are are talking about my merged change preventing using
Py_TYPE() as an l-value, this is out of the scope of the PEP on
purpose.

Py_TYPE(), Py_REFCNT() and Py_SIZE() could be used an l-value in
Python 3.9, but it's no longer the case in Python 3.11. Apart of that,
I'm not aware of other macros which could be "abused" as l-value.


Wait, so this PEP is about converting macros to functions, but not about 
converting Py_SIZE to a function? I'm confused. Why is Py_SIZE listed in 
the PEP?




There are macros which can be "abused" ("used") to access to structure
members and object internals. For example, &PyTuple_GET_ITEM(tuple, 0)
and &PyList_GET_ITEM(list, 0) can be "abused" to access directly to an
array of PyObject* (PyObject** type) and so modify directly a
tuple/list. I would like to change that (disallow it), but it's out of
the scope of the PEP. See https://bugs.python.org/issue41078 for my
previous failed attempt (it broke too many things). But this is more
in the scope of the PEP 620 which is a different PEP. >


Are there any other issues that break existing code?


I listed all known backward incompatibles changes in the Backward
Compatibility section. I'm not aware of other backward incompatible
changes caused by the PEP.

Converting macros to static inline functions or regular functions
didn't change the API for the macros already converted, the ones
listed in the PEP.


It did for e.g. Py_SIZE, which no longer behaves like in 3.9, nor as it 
was documented in 3.8: 
https://docs.python.org/3.8/c-api/structures.html#c.Py_SIZE
Yet Py_SIZE is listed in the PEP as "Macros converted to static inline 
functions", so clearly it is in scope.

Same for Py_TYPE. Are there others?



The "Cast to PyObject*" section talks about adding new private functions
like _Py_TYPE, which are type-safe, but keeping old names (like Py_TYPE)
as macros that do a cast.
Could the newly added functions be made public from the start? (They
could use names like Py_Type.) This would allow languages that don't
have macros to use them directly, and if the non-typesafe macros are
ever discouraged/deprecated/removed, this would allow writing compatible
code now.


I don't want to increase the size of the C API and so I chose to make
the inner function accepting PyObject* private.

I see the addition of an hypothetical Py_Type() function as an
increase of the maintenance burden: we would have to maintain it,
document it, maybe add it to the limited C API / stable ABI, write
tests, etc.

I prefer to restrict the scope of the PEP. If you want to add variants
only accepting PyObject*, that's fine, but I suggest to open a
separated issue / PEP. Also, it can be discussed on a case by case
basic (function per function).


Since functions like _Py_TYPE will need to be maintained as part of the 
stable ABI, I'd like to do this right from the start. If you don't, can 
you add this to Rejected ideas?



I'm still interested in:

Since this is about converting existing macros (and not writing new ones), can you talk 
about which of the "macro pitfalls" apply to the macros in CPython that 
were/will be changed?

Is that just a theoretical issue?

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/K52XIAU4WBVSJBAXZLC4EOP3BBCILLN

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Victor Stinner 
On Wed, Nov 24, 2021 at 10:59 AM Petr Viktorin  wrote:
> Since this is about converting existing macros (and not writing new
> ones), can you talk about which of the "macro pitfalls" apply to the
> macros in CPython that were/will be changed?

The PEP 670 lists many pitfalls affecting existing macros. Some
pitfalls are already worked around in the current implementations, but
the point is that it's easy to miss pitfalls when reviewing code
adding new macros or modifying macros.

Erlend did an analysis in: https://bugs.python.org/issue43502

For macros reusing arguments (known as "Duplication of side effects"
in GCC Macro Pitfalls), see his list:
https://bugs.python.org/file49877/macros-that-reuse-args.txt

Victor
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/25RWEFLDPP7I7Y7DUT75YBRVMMSNN5QI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Victor Stinner 
On Wed, Nov 24, 2021 at 2:18 PM Petr Viktorin  wrote:
> >> The "Backwards Compatibility" section is very small. Can you give a list
> >> of macros which lost/will lose "return values"?
> >
> > https://bugs.python.org/issue45476 lists many of them. See also:
> > https://github.com/python/cpython/pull/28976
>
> Also, this PR is about preventing the use of some macros as l-values,
> which you say is out of scope for the PEP. I'm connfused.

Oh right, now I'm also confused :-) I forgot about the details.

"Py_TYPE(obj) = new_type;" was used in 3rd party C extensions when
defining static types to work around linker issues on Windows.
Changing Py_TYPE() to disallow using it as an l-value is an
incompatible change.

>From what I saw in bpo-45476, the functions that I propose to change
are not used as l-value. Technically, it's an incompatible change. In
practice, it should not impact any 3rd party project.

For example, PyFloat_AS_DOUBLE() is used to read a float value (ex:
"double x = PyFloat_AS_DOUBLE(obj);"), but not to set a float value
(ex: "PyFloat_AS_DOUBLE(obj) = 1.0;").

Ok, I should clarify that in the PEP.


> Wait, so this PEP is about converting macros to functions, but not about
> converting Py_SIZE to a function? I'm confused. Why is Py_SIZE listed in
> the PEP?

Py_SIZE() is already converted to a static inline function. Later, it
can be converted to a regular function if it makes sense.

It's listed in the PEP to show macros which are already converted, to
help to estimate how many 3rd party applications would be affected by
the PEP.

Py_REFCNT(), Py_TYPE() and Py_SIZE() are special because they were
used as l-value on purpose. As far as I know, they were the only 3
macros used as l-value, no?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KJDXNWRDQ36R72XOZII32K54POAGT75Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Petr Viktorin 




On 24. 11. 21 15:22, Victor Stinner wrote:

On Wed, Nov 24, 2021 at 10:59 AM Petr Viktorin  wrote:

Since this is about converting existing macros (and not writing new
ones), can you talk about which of the "macro pitfalls" apply to the
macros in CPython that were/will be changed?


The PEP 670 lists many pitfalls affecting existing macros. Some
pitfalls are already worked around in the current implementations, but
the point is that it's easy to miss pitfalls when reviewing code
adding new macros or modifying macros.

Erlend did an analysis in: https://bugs.python.org/issue43502

For macros reusing arguments (known as "Duplication of side effects"
in GCC Macro Pitfalls), see his list:
https://bugs.python.org/file49877/macros-that-reuse-args.txt


That's s nice list. Could you link to it in the PEP, so the next person 
won't have to ask?



Meanwhile, I think I found a major source of my confusion with the PEP: 
I'm not clear on what it actually proposes. Is it justification for 
changes that were already done, or a plan for more changes, or a policy 
change ("don't write a public macro if it can be a function"), or all of 
those?

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/AMZ4Z45JOYIS6YRMAFGWAN4D45WMO4DO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Petr Viktorin 




On 24. 11. 21 15:32, Victor Stinner wrote:

On Wed, Nov 24, 2021 at 2:18 PM Petr Viktorin  wrote:

The "Backwards Compatibility" section is very small. Can you give a list
of macros which lost/will lose "return values"?


https://bugs.python.org/issue45476 lists many of them. See also:
https://github.com/python/cpython/pull/28976


Also, this PR is about preventing the use of some macros as l-values,
which you say is out of scope for the PEP. I'm connfused.


Oh right, now I'm also confused :-) I forgot about the details.

"Py_TYPE(obj) = new_type;" was used in 3rd party C extensions when
defining static types to work around linker issues on Windows.
Changing Py_TYPE() to disallow using it as an l-value is an
incompatible change.

 From what I saw in bpo-45476, the functions that I propose to change
are not used as l-value. Technically, it's an incompatible change. In
practice, it should not impact any 3rd party project.

For example, PyFloat_AS_DOUBLE() is used to read a float value (ex:
"double x = PyFloat_AS_DOUBLE(obj);"), but not to set a float value
(ex: "PyFloat_AS_DOUBLE(obj) = 1.0;").

Ok, I should clarify that in the PEP.


Yes. *Each* incompatible change should be listed, even if you believe it 
won't affect any project. The PEP reader should be allowed to judge if 
your assumptions are correct.


e.g. I've seen projects actually use "Py_TYPE(obj) = new_type;" to 
change an object's type after it was given to Python code. It would be 
great to document why that's wrong *and* what to do instead, both in the 
PEP that introduced the change and in the "What's New" entry.






Wait, so this PEP is about converting macros to functions, but not about
converting Py_SIZE to a function? I'm confused. Why is Py_SIZE listed in
the PEP?


Py_SIZE() is already converted to a static inline function. Later, it
can be converted to a regular function if it makes sense.

It's listed in the PEP to show macros which are already converted, to
help to estimate how many 3rd party applications would be affected by
the PEP.


Is such an estimate available?



Py_REFCNT(), Py_TYPE() and Py_SIZE() are special because they were
used as l-value on purpose. As far as I know, they were the only 3
macros used as l-value, no?


Who knows? If there's a list of what to change, someone can go through 
it and answer this for each macro.

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/XBEZS3KXDFGVEZMAO6HJGFB5YSYND7LS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Victor Stinner 
I ran the Python test suite to compare macros versus static inline
functions (using PR 29728). I built Python with gcc -O3, LTO and PGO
optimizations.

=> There is *no* significant performance difference.

I understand that static inline functions are inlined by the C
compiler (GCC) as expected.

* Macros: 361 sec +- 1 sec
* Static inline functions: 361 sec +- 1 sec

$ python3 -m pyperf compare_to pgo-lto_test_suite_macros.json
pgo-lto_test_suite_static_inline.json
Benchmark hidden because not significant (1): command

I built Python with:

$ ./configure --with-lto --enable-optimizations --prefix $PWD/install
$ taskset --cpu-list 2,3,6,7 make
$ make install

And I ran the following benchmark, run the test suite 5 times using
pyperf which pin the process to isolated CPUs:

$ python3 -m pyperf command -p1 --warmups=0 --loops=1 --values=5 -v -o
../pgo-lto_test_suite_macros.json -- ./bin/python3.11 -m test -j5

I isolated 4 logical CPUs (2, 3, 6 and 7) on 8: physical CPU cores 2
and 3 (cores 0 and 1 are not isolated).

--

Right now, I cannot use pyperformance: it fails to create a virtual
environment because greenlet fails to build with Python 3.11. On
speed.python.org, the benchmark are still running only because...
pyperformance uses a cached binary wheel of greenlet. It looks
dangerous to use a cached wheel, since the Python ABI (PyThreadState)
changed!

Help is welcomed to repair pyperformance:
https://github.com/python/pyperformance/issues/113

Victor

On Wed, Nov 24, 2021 at 12:27 AM Guido van Rossum  wrote:
>
> On Tue, Nov 23, 2021 at 3:15 PM Antoine Pitrou  wrote:
>>
>> On Tue, 23 Nov 2021 18:00:28 +0100
>> Victor Stinner  wrote:
>>
>> > I didn't run benchmarks on Python built in release mode, since gcc -O3
>> > with LTO and PGO should inline all static inline functions and I don't
>> > expect any difference between macros and static inline functions.
>>
>> That would actually be interesting, since there can be surprises
>> sometimes with compilers... (function inlining depends on heuristics,
>> for example, and there may be positive or negative interactions with
>> other optimizations)
>
>
> Thanks Antoine. We definitely need to push back on such "expectations" and 
> turn them into facts by performing careful measurements. Surprises lurk 
> everywhere. See e.g. 
> https://github.com/faster-cpython/ideas/issues/109#issuecomment-975619113 
> (and watch the Emery Berger video linked there if you haven't already).
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/[email protected]/message/TPUARSPZ7MLDHHWZKPO3FLMIEHMOM6SB/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/NINSZIW2LU2AEL2FB6FWXV3EA7BA7XMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Terry Reedy 

On 11/23/2021 6:21 PM, Guido van Rossum wrote:

Thanks Antoine. We definitely need to push back on such "expectations" 
and turn them into facts by performing careful measurements. Surprises 
lurk everywhere. See e.g. 
https://github.com/faster-cpython/ideas/issues/109#issuecomment-975619113  
(and watch the Emery Berger video linked there if you haven't already).


Surprises indeed.  When I discussed this with my daughter after watching 
it, she told me that the Sims 2 with multiple mods and multiple player 
characters, the game loaded very slowly, taking several minute on 
machines of the time.  Some players discovered that it loaded minutes 
faster if player character names and corresponding filenames were 
limited to a subset of printable ascii chars (no space, %, and some 
others).  Someone even wrote a renamer program.


Has Python on linux been run with with the cos program yet?


--
Terry Jan Reedy

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/3QCWTG3QIHHCVHHOTWXMZJL4ANWI6H2I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 670: Convert macros to functions in the Python C API

2021-11-24 Thread Guido van Rossum 
Brandt looked at coz for Python but it didn't seem to find anything useful
-- it singled out random lines in the code. :-(

On Wed, Nov 24, 2021 at 10:13 AM Terry Reedy  wrote:

> On 11/23/2021 6:21 PM, Guido van Rossum wrote:
>
> > Thanks Antoine. We definitely need to push back on such "expectations"
> > and turn them into facts by performing careful measurements. Surprises
> > lurk everywhere. See e.g.
> >
> https://github.com/faster-cpython/ideas/issues/109#issuecomment-975619113
> 
>
> > (and watch the Emery Berger video linked there if you haven't already).
>
> Surprises indeed.  When I discussed this with my daughter after watching
> it, she told me that the Sims 2 with multiple mods and multiple player
> characters, the game loaded very slowly, taking several minute on
> machines of the time.  Some players discovered that it loaded minutes
> faster if player character names and corresponding filenames were
> limited to a subset of printable ascii chars (no space, %, and some
> others).  Someone even wrote a renamer program.
>
> Has Python on linux been run with with the cos program yet?
>
>
> --
> Terry Jan Reedy
>
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/3QCWTG3QIHHCVHHOTWXMZJL4ANWI6H2I/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/KRBWNQCJGPC2Z4LWQKXHTM4VB4JVG3YB/
Code of Conduct: http://python.org/psf/codeofconduct/