[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Petr Viktorin]

On 06. 12. 21 21:50, Guido van Rossum wrote:
On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin > wrote:


On 06. 12. 21 20:29, Guido van Rossum wrote:
 > Hi Petr,
 >
 > In PEP 384 it is written that no functions starting with an
underscore
 > are part of the stable ABI:
 >
 > PEP 384 -- Defining a Stable ABI | Python.org
 > >
 >  > All functions starting with _Py are not available to applications
 >
 > OTOH there's a data file in the repo, Misc/stabe_abi.txt, which
lists
 > many functions starting with _Py_, for example
_PyObject_GC_Malloc. Then
 > again, that function is not listed in Doc/data/stable_abi.dat. (I
didn't
 > check other functions, but maybe there are others.)
 >
 > So is Misc/stable_abi.txt just out of date? Or how can the
discrepancy
 > be explained?

These are not part of the limited API, so extension authors can't use
them in the C source. But they typically are (or have been) called by
macros from the limited API. So, they are part of the stable ABI; they
need to be exported.

Misc/stable_abi.txt says "abi_only" for all of these. They don't
show up
in the user-facing docs.


Thanks, that helps. It's too bad that there's no comment at the top 
explaining the format (in fact it appears to discourage reading the file?).


You can read it, but I want to discourage people from relying on the 
format: Tools/scripts/stable_abi.py should be the only consumer.

I will add a comment though.


Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc 
from stable_abi.txt in https://github.com/python/cpython/pull/29879 
 Is that allowed? If it's 
being used by a macro it means code using that macro will fail unless 
recompiled for 3.11.


Generally, that's not allowed. In this particular case, Victor's 
analysis is right: if you trawl through the history from 3.2 on, you can 
see that you can't call _PyObject_GC_Malloc via macros in the limited 
API. So yes, this one can be removed.


I'll also note that removing things that are "allowed" to go is not nice 
to people who relied on PEP 384, which says that defining Py_LIMITED_API 
"will hide all definitions that are not part of the ABI" -- even though 
that's incompatible with the part where it says "All functions starting 
with _Py are not available to applications".
PEP 384 is a historical document, but before 3.10 it was the best 
available documentation. PEP 652 sort of changed the rules mid-course 
(ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility).



But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it. 
Go ahead and remove it.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S7PL37E4Z4RHICOHTK32NCQTYN5C6FAQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Petr Viktorin]

On 30. 11. 21 19:52, Victor Stinner wrote:

On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum  wrote:

How about *not* asking for an exception and just following the PEP 387 process? 
Is that really too burdensome?


The Backward Compatibility section gives an explanation:

"This change does not follow the PEP 387 deprecation process. There is no
known way to emit a deprecation warning when a macro is used as a
l-value, but not when it's used differently (ex: r-value)."

Apart of compiler warnings, one way to implement the PEP 387
"deprecation process" would be to announce the change in two "What's
New in Python 3.X?" documents. But I expect that it will not be
efficient. Extract of the Rejected Idea section:

"(...) only few developers read the documentation, and only a minority
is tracking changes of the Python C API documentation."

In my experience, even if a DeprecationWarning is emitted at runtime,
developers miss or ignore it. See the recent "[Python-Dev] Do we need
to remove everything that's deprecated?" discussion and complains
about recent removal of deprecated features, like:

* collections.MutableMapping was deprecated for 7 Python versions
(deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11
* the "U" open() flag was deprecated for 10 Python versions
(deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11

For this specific PEP changes, I consider that the number of impacted
projects is low enough to skip a deprecation process: only 4 projects
are known to be impacted. One year ago (Python 3.10), 16 were
impacted, and 12 have already been updated in the meanwhile. I'm
talking especially about Py_TYPE() and Py_SIZE() changes which, again,
has been approved by the Steering Council.



The current version of the PEP looks nice, but I don't think the 
rationale is strong enough.

I believe we should:
- Mark the l-value usage as deprecated in the docs,
- And then do nothing until we find an actual case where this issue 
blocks development (or is actively dangerous for users).


Specifically, for each of the Rationale parts:


## Using a macro as a l-value

The practice was often discouraged (e.g. by GET in many of the names), 
or even relatively widely and successfully used (e.g. Py_TYPE before 
Python 3.9).


If we would deprecate using Py_REFCNT as l-value in the docs, but wait 
with the conversion until it was *actually* needed, we would not lose 
anything:
- Users would *still* have no period of visible compiler warnings or 
DeprecationWarning.
- There would be more time for users to react to the documentation 
warning. Or even come up with a linter, or try compiling their favorite 
extensions with HPy/nogil and fixing the issues *on their own schedule*.



## CPython nogil fork

In CPython, we cannot change structs that are part of the stable ABI -- 
such as PyObject.ob_refcnt. IMO, getting rid of the macros that access 
ob_refcnt is a relatively small part of this issue.


AFAICS, the technical change is trivial compared to nogil, and can be 
easily made in the nogil fork -- if it is actually necessary in the end.



## HPy project

There is no reason for "Searching and replacing Py_SET_SIZE()".
If this change was not made, and Py_SIZE was semi-mechanically replaced 
by HPy_Length(), then misuses of Py_SIZE (using is as l-value) can be 
detected very easily: just compile against HPy.
And if the autoreplacer is smart enough to see when it should use 
HPyTupleBuilder/HPyListBuilder, then it can surely see when Py_SIZE is 
(mis)used as l-value!


There will always be some changes necessary when porting extensions to HPy.
CPython should definitely indicate what the best practice is, so that 
HPY adopters have an easy time convincing projects to take their pull 
requests -- but it should not break code for people who don't care about 
HPy yet.



## GraalVM Python

This PEP is not enough to get rid of wrappers in GraalVM, yet it forces 
users of CPython to adapt. Is it a part of a *plan* to remove wrappers, 
or just a minor step in what looks like the general direction?
I do agree this PEP looks like a good step towards a long-term goal. But 
even so, it should be made when it *actually benefits* existing users, 
or allows some concrete good thing -- an optimization, a more 
maintainable implementation, something that outweighs the need for churn 
in code that worked up to now.




Overall, the Rationale seems hollow to me. It's breaking existing code 
-- however bad that code is -- in the name of ideals rather than 
concrete improvements.
Until disallowing macros as l-values allows concrete improvements in 
CPython, it should be the job of linters.



FWIW, I do encourage alternative implementations to just not support 
l-value macros. There are only few projects doing this, and the fix is 
often (but not always!) easy. This should be a very small part of 
porting something to a different Python implementation (but I could 
def

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Antoine Pitrou]

On Tue, 7 Dec 2021 15:39:25 +0100
Petr Viktorin  wrote:

> On 30. 11. 21 19:52, Victor Stinner wrote:
> > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum  wrote:  
> >> How about *not* asking for an exception and just following the PEP 387 
> >> process? Is that really too burdensome?  
> > 
> > The Backward Compatibility section gives an explanation:
> > 
> > "This change does not follow the PEP 387 deprecation process. There is no
> > known way to emit a deprecation warning when a macro is used as a
> > l-value, but not when it's used differently (ex: r-value)."
> > 
> > Apart of compiler warnings, one way to implement the PEP 387
> > "deprecation process" would be to announce the change in two "What's
> > New in Python 3.X?" documents. But I expect that it will not be
> > efficient. Extract of the Rejected Idea section:
> > 
> > "(...) only few developers read the documentation, and only a minority
> > is tracking changes of the Python C API documentation."
> > 
> > In my experience, even if a DeprecationWarning is emitted at runtime,
> > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > to remove everything that's deprecated?" discussion and complains
> > about recent removal of deprecated features, like:
> > 
> > * collections.MutableMapping was deprecated for 7 Python versions
> > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > removed again in 3.11
> > * the "U" open() flag was deprecated for 10 Python versions
> > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > removed again in 3.11
> > 
> > For this specific PEP changes, I consider that the number of impacted
> > projects is low enough to skip a deprecation process: only 4 projects
> > are known to be impacted. One year ago (Python 3.10), 16 were
> > impacted, and 12 have already been updated in the meanwhile. I'm
> > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > has been approved by the Steering Council.  
> 
> 
> The current version of the PEP looks nice, but I don't think the 
> rationale is strong enough.
> I believe we should:
> - Mark the l-value usage as deprecated in the docs,
> - And then do nothing until we find an actual case where this issue 
> blocks development (or is actively dangerous for users).

Is there a way to emit a compilation warning when those macros are used
as l-values? Even if only enabled on some compilers.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 669: Low Impact Monitoring for CPython

[Mark Shannon]

Hi,

I would like to announce latest PEP, PEP 669: Low Impact Monitoring for CPython.

The aim of this PEP is to provide an API for profilers, debuggers and other 
tools to avoid the punitive overhead of using sys.settrace.

If you have any interest in profilers, debuggers, coverage tools or anything of 
that ilk, then do please take a look.

There is no change to the language and it adds 7 functions to the sys module, 
so shouldn't be too intrusive for those of who aren't planning on implementing 
any of those tools.

As always, all feedback and comments are welcome.

You can read the PEP here:
https://python.github.io/peps/pep-0669/

Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VNSD4TSAM2BM64FJNIQPAOPNEGNX4MDX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 669: Low Impact Monitoring for CPython

[Victor Stinner]

Interesting! Some remarks about the proposed API.

On Tue, Dec 7, 2021 at 4:58 PM Mark Shannon  wrote:
> There is no change to the language and it adds 7 functions to the sys module, 
> so shouldn't be too intrusive for those of who aren't planning on 
> implementing any of those tools.

Where are event constants defined, like PY_CALL?

Would it make sense to add a small module for these event constants
and functions?

Would it be possible to define enums (rather than simple ints) for
these event constants, to ease debugging?

sys.get_monitoring_events()->int
sys.set_monitoring_events(event_set: int)
sys.get_local_monitoring_events(code: CodeType)->int
sys.set_local_monitoring_events(code: CodeType, event_set: int)
sys.register_monitoring_callback(event: int, func: Callable)
sys.insert_marker(code: CodeType, offset: int, marker_id=0: range(256))
sys.remove_marker(code: CodeType, offset: int)

For example:

monitoring_events.get_events()->int
monitoring_events.set_events(event_set: int)
monitoring_events.get_local_events(code: CodeType)->int
monitoring_events.set_local_events(code: CodeType, event_set: int)
monitoring_events.register_callback(event: int, func: Callable)
monitoring_events.insert_marker(code: CodeType, offset: int,
marker_id=0: range(256))
monitoring_events.remove_marker(code: CodeType, offset: int)
monitoring_events.PY_CALL
(... other constants ...)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OBL7T7IN6P2YOLMPFPVDCU6JELLOUDOE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Joao S. O. Bueno]

Sorry for stepping in - but I am seeing too many arguments in favour
of the rules because "they are the rules", and just Victor arguing with
what is met in the "real world".

But if this update can be done by a simple search/replace on the C source
of projects,
I can only perceive two scenarios this will affect: well maintained
projects,
 for which it is fixable in minutes, and  stale packages, no longer
released
that "happen to work" when someone downloads and builds for new
Python versions. In these cases, the build will fail. If the person trying
the build can't fix it, but can take the error to a proper, or high
visibility,
forum, someone will be able to come to the fix, leading to renewed
visibility for the otherwise stale package.



On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou  wrote:

> On Tue, 7 Dec 2021 15:39:25 +0100
> Petr Viktorin  wrote:
>
> > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum 
> wrote:
> > >> How about *not* asking for an exception and just following the PEP
> 387 process? Is that really too burdensome?
> > >
> > > The Backward Compatibility section gives an explanation:
> > >
> > > "This change does not follow the PEP 387 deprecation process. There is
> no
> > > known way to emit a deprecation warning when a macro is used as a
> > > l-value, but not when it's used differently (ex: r-value)."
> > >
> > > Apart of compiler warnings, one way to implement the PEP 387
> > > "deprecation process" would be to announce the change in two "What's
> > > New in Python 3.X?" documents. But I expect that it will not be
> > > efficient. Extract of the Rejected Idea section:
> > >
> > > "(...) only few developers read the documentation, and only a minority
> > > is tracking changes of the Python C API documentation."
> > >
> > > In my experience, even if a DeprecationWarning is emitted at runtime,
> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > > to remove everything that's deprecated?" discussion and complains
> > > about recent removal of deprecated features, like:
> > >
> > > * collections.MutableMapping was deprecated for 7 Python versions
> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > > * the "U" open() flag was deprecated for 10 Python versions
> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > >
> > > For this specific PEP changes, I consider that the number of impacted
> > > projects is low enough to skip a deprecation process: only 4 projects
> > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > impacted, and 12 have already been updated in the meanwhile. I'm
> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > > has been approved by the Steering Council.
> >
> >
> > The current version of the PEP looks nice, but I don't think the
> > rationale is strong enough.
> > I believe we should:
> > - Mark the l-value usage as deprecated in the docs,
> > - And then do nothing until we find an actual case where this issue
> > blocks development (or is actively dangerous for users).
>
> Is there a way to emit a compilation warning when those macros are used
> as l-values? Even if only enabled on some compilers.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT3VK6REJ3BEBVGNDGZCIXWFCA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Guido van Rossum]

On Tue, Dec 7, 2021 at 12:58 AM Petr Viktorin  wrote:

> On 06. 12. 21 21:50, Guido van Rossum wrote:
> > On Mon, Dec 6, 2021 at 12:12 PM Petr Viktorin  > > wrote:
> >
> > On 06. 12. 21 20:29, Guido van Rossum wrote:
> >  > Hi Petr,
> >  >
> >  > In PEP 384 it is written that no functions starting with an
> > underscore
> >  > are part of the stable ABI:
> >  >
> >  > PEP 384 -- Defining a Stable ABI | Python.org
> >  >  > >
> >  >  > All functions starting with _Py are not available to
> applications
> >  >
> >  > OTOH there's a data file in the repo, Misc/stabe_abi.txt, which
> > lists
> >  > many functions starting with _Py_, for example
> > _PyObject_GC_Malloc. Then
> >  > again, that function is not listed in Doc/data/stable_abi.dat. (I
> > didn't
> >  > check other functions, but maybe there are others.)
> >  >
> >  > So is Misc/stable_abi.txt just out of date? Or how can the
> > discrepancy
> >  > be explained?
> >
> > These are not part of the limited API, so extension authors can't use
> > them in the C source. But they typically are (or have been) called by
> > macros from the limited API. So, they are part of the stable ABI;
> they
> > need to be exported.
> >
> > Misc/stable_abi.txt says "abi_only" for all of these. They don't
> > show up
> > in the user-facing docs.
> >
> >
> > Thanks, that helps. It's too bad that there's no comment at the top
> > explaining the format (in fact it appears to discourage reading the
> file?).
>
> You can read it, but I want to discourage people from relying on the
> format: Tools/scripts/stable_abi.py should be the only consumer.
> I will add a comment though.
>

I don't mind that the format might change. But I feel that I should be able
to understand the current format so I can know how to maintain the document
without using cargo-culting (and trying to understand what the tool does is
not an option). So thanks for adding the comment.


> > Also, it looks like Mark is proposing to *remove* _PyObject_GC_Malloc
> > from stable_abi.txt in https://github.com/python/cpython/pull/29879
> >  Is that allowed? If it's
> > being used by a macro it means code using that macro will fail unless
> > recompiled for 3.11.
>
> Generally, that's not allowed. In this particular case, Victor's
> analysis is right: if you trawl through the history from 3.2 on, you can
> see that you can't call _PyObject_GC_Malloc via macros in the limited
> API. So yes, this one can be removed.
>

Okay, that's very subtle, so thanks for confirming.


> I'll also note that removing things that are "allowed" to go is not nice
> to people who relied on PEP 384, which says that defining Py_LIMITED_API
> "will hide all definitions that are not part of the ABI" -- even though
> that's incompatible with the part where it says "All functions starting
> with _Py are not available to applications".
>

I don't actually really follow what you are trying to say here. Probably
because I've never paid much attention to PEP 384. I guess the API is
confusing because the "right" way to do it (having to define some symbol to
*expose* extra stuff rather than to *hide* stuff) was not possible for
backwards compatibility reasons. But the extra negative will forever make
this confusing. Also, "All functions starting with _Py are not available"
sounds like a clumsy way to say "No functions starting with _Py are
available" (and you left out whether Py_LIMITED_API affects that
availability, whether it was intended to affect it, whether it did in
practice affect it in all cases, etc.

I assume it would be insensitive to ask whether we could just get rid of
the stable ABI altogether and focus on the limited API? Just tell everyone
they have to rebuild binary wheels for every Python feature release.
Presumably the deprecation of the stable ABI itself would require a
two-release waiting period. But maybe it would be worth it, given how
subtle it is to do the historical research about even a single function.


> PEP 384 is a historical document, but before 3.10 it was the best
> available documentation. PEP 652 sort of changed the rules mid-course
> (ref. https://www.python.org/dev/peps/pep-0652/#backwards-compatibility).
>
>
> But for _PyObject_GC_Malloc specifically, IMO the speedup is worth it.
> Go ahead and remove it.
>

It's gone. Mark landed his change earlier today.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https:

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Sebastian Berg]

On Tue, 2021-12-07 at 13:54 -0300, Joao S. O. Bueno wrote:
> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing
> with
> what is met in the "real world".
> 
> But if this update can be done by a simple search/replace on the C
> source
> of projects,
> I can only perceive two scenarios this will affect: well maintained
> projects,
>  for which it is fixable in minutes, and  stale packages, no longer
> released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person
> trying
> the build can't fix it, but can take the error to a proper, or high
> visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
> 

The problem are really less-maintained projects that may miss it or
take very long to react.  And that may be very frustrating for their
users (who will not have a work-around beyond patching the project!).

So the question is whether these are so few users, that it is OK to
break them (even many bug fixes will break someone, after all).
The other consideration may be that documenting the change for a 1-2
years may achieve almost nothing except frustrating Victor ;).


One thing we once did in NumPy (for a runtime problem), was to
intentionally break everyone at pre-release/dev time to point out what
code needed fixing.  Then flip the switch back at release time as to
not break production.
After a long enough time we enabled it for release mode.

Not saying that it was nice, but it was the only alternative would have
been to never fix it.

A similar switch could be worthwhile if it helps Victor with
experimenting on the dev-branch or reach a useful amount of projects.
Of course, I am not sure it would do either...

Cheers,

Sebastian



> 
> 
> On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou 
> wrote:
> 
> > On Tue, 7 Dec 2021 15:39:25 +0100
> > Petr Viktorin  wrote:
> > 
> > > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum
> > > > 
> > wrote:
> > > > > How about *not* asking for an exception and just following
> > > > > the PEP
> > 387 process? Is that really too burdensome?
> > > > 
> > > > The Backward Compatibility section gives an explanation:
> > > > 
> > > > "This change does not follow the PEP 387 deprecation process.
> > > > There is
> > no
> > > > known way to emit a deprecation warning when a macro is used as
> > > > a
> > > > l-value, but not when it's used differently (ex: r-value)."
> > > > 
> > > > Apart of compiler warnings, one way to implement the PEP 387
> > > > "deprecation process" would be to announce the change in two
> > > > "What's
> > > > New in Python 3.X?" documents. But I expect that it will not be
> > > > efficient. Extract of the Rejected Idea section:
> > > > 
> > > > "(...) only few developers read the documentation, and only a
> > > > minority
> > > > is tracking changes of the Python C API documentation."
> > > > 
> > > > In my experience, even if a DeprecationWarning is emitted at
> > > > runtime,
> > > > developers miss or ignore it. See the recent "[Python-Dev] Do
> > > > we need
> > > > to remove everything that's deprecated?" discussion and
> > > > complains
> > > > about recent removal of deprecated features, like:
> > > > 
> > > > * collections.MutableMapping was deprecated for 7 Python
> > > > versions
> > > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9
> > > > beta,
> > > > removed again in 3.11
> > > > * the "U" open() flag was deprecated for 10 Python versions
> > > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9
> > > > beta,
> > > > removed again in 3.11
> > > > 
> > > > For this specific PEP changes, I consider that the number of
> > > > impacted
> > > > projects is low enough to skip a deprecation process: only 4
> > > > projects
> > > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > > impacted, and 12 have already been updated in the meanwhile.
> > > > I'm
> > > > talking especially about Py_TYPE() and Py_SIZE() changes which,
> > > > again,
> > > > has been approved by the Steering Council.
> > > 
> > > 
> > > The current version of the PEP looks nice, but I don't think the
> > > rationale is strong enough.
> > > I believe we should:
> > > - Mark the l-value usage as deprecated in the docs,
> > > - And then do nothing until we find an actual case where this
> > > issue
> > > blocks development (or is actively dangerous for users).
> > 
> > Is there a way to emit a compilation warning when those macros are
> > used
> > as l-values? Even if only enabled on some compilers.
> > 
> > Regards
> > 
> > Antoine.
> > 
> > 
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/p

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Christian Heimes]

On 07/12/2021 19.28, Guido van Rossum wrote:
I assume it would be insensitive to ask whether we could just get rid of 
the stable ABI altogether and focus on the limited API? Just tell 
everyone they have to rebuild binary wheels for every Python feature 
release. Presumably the deprecation of the stable ABI itself would 
require a two-release waiting period. But maybe it would be worth it, 
given how subtle it is to do the historical research about even a single 
function.


The stable ABI is useful for Python packages that ship binary wheels.

Take PyCA cryptography [1] as an example. Alex and Paul already build, 
upload, and ship 12 abi3 wheels for each release and combinations of CPU 
arch, platform, and libc ABI. Without a stable ABI they would have to 
create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The 
number will only increase over time. Python 3.6 is very common on 
LTS/Enterprise Linux distros.


If the current stable ABI makes performance improvements too complex 
then we should consider to define a new stable ABI with less symbols.


Christian

[1] https://pypi.org/project/cryptography/#files
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PQ7GNBTFAHIC6HWNZ62MQJJOV75N7UAT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Guido van Rossum]

On Tue, Dec 7, 2021 at 9:00 AM Joao S. O. Bueno 
wrote:

> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing with
> what is met in the "real world".
>
> But if this update can be done by a simple search/replace on the C source
> of projects,
> I can only perceive two scenarios this will affect: well maintained
> projects,
>  for which it is fixable in minutes, and  stale packages, no longer
> released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person trying
> the build can't fix it, but can take the error to a proper, or high
> visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
>

It sounds like you (like probably many others) don't quite understand why
"the rules" exist.

Please trust me that what you perceive as simple updates, is not so simple
when you take the whole ecosystem of dependencies into account. You may
recall the long transition from Python 2 to 3. This was so painful in part
because the core dev team (led by myself) was similarly optimistic about
"oh, people can easily fix the few small things that will crop up". In
practice, things like this take a very long time to fix everywhere, as
people are waiting for their dependencies to solve the issue first before
they can even start testing, and so on. (For example, many libraries still
don't have wheels for Python 3.10, even though it was released over two
months ago and 3.10.1 is already out.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TL3ZA3MSVFL6HFAC34SNRKU46B2VKDEB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Clarification regarding Stable ABI and _Py_*

[Guido van Rossum]

On Tue, Dec 7, 2021 at 11:02 AM Christian Heimes 
wrote:

> On 07/12/2021 19.28, Guido van Rossum wrote:
> > I assume it would be insensitive to ask whether we could just get rid of
> > the stable ABI altogether and focus on the limited API? Just tell
> > everyone they have to rebuild binary wheels for every Python feature
> > release. Presumably the deprecation of the stable ABI itself would
> > require a two-release waiting period. But maybe it would be worth it,
> > given how subtle it is to do the historical research about even a single
> > function.
>
> The stable ABI is useful for Python packages that ship binary wheels.
>
> Take PyCA cryptography [1] as an example. Alex and Paul already build,
> upload, and ship 12 abi3 wheels for each release and combinations of CPU
> arch, platform, and libc ABI. Without a stable ABI they would have to
> create a total of 60 binary abi3 wheels for Python 3.6 to 3.10. The
> number will only increase over time. Python 3.6 is very common on
> LTS/Enterprise Linux distros.
>

Thanks, that's a very useful example.


> If the current stable ABI makes performance improvements too complex
> then we should consider to define a new stable ABI with less symbols.
>

But then we will run into backwards compatibility concerns. Suppose we want
to delete *one* functions from the stable ABI. How many releases do we have
to wait before we can actually delete (as opposed to just deprecate) it? It
sounds like you're saying it would take 5 releases, i.e. if we deprecate it
in 3.11, we can delete it in 3.16. It would probably be easier to just not
bother with the deprecation.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DITJQNFGU5LFC3JRQNTZ3OFE2GDESKTH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Steve Dower]

On 12/7/2021 6:52 PM, Sebastian Berg wrote:

One thing we once did in NumPy (for a runtime problem), was to
intentionally break everyone at pre-release/dev time to point out what
code needed fixing.  Then flip the switch back at release time as to
not break production.
After a long enough time we enabled it for release mode.

Not saying that it was nice, but it was the only alternative would have
been to never fix it.


I like this idea. We'd have to turn it back for RC, and ensure that it's 
possible to have working code both before/after the change. We may be 
getting enough usage during beta for it to be worthwhile, though we 
still have the problem of knock-on effects (where e.g. until NumPy 
works, nothing that depends on it can even begin testing).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQ7KF656QIHPKLT5Z23P3W5OYOGH4J5K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Guido van Rossum]

On Tue, Dec 7, 2021 at 1:09 PM Steve Dower  wrote:

> On 12/7/2021 6:52 PM, Sebastian Berg wrote:
> > One thing we once did in NumPy (for a runtime problem), was to
> > intentionally break everyone at pre-release/dev time to point out what
> > code needed fixing.  Then flip the switch back at release time as to
> > not break production.
> > After a long enough time we enabled it for release mode.
> >
> > Not saying that it was nice, but it was the only alternative would have
> > been to never fix it.
>
> I like this idea. We'd have to turn it back for RC, and ensure that it's
> possible to have working code both before/after the change. We may be
> getting enough usage during beta for it to be worthwhile, though we
> still have the problem of knock-on effects (where e.g. until NumPy
> works, nothing that depends on it can even begin testing).
>

Yeah, this sounds like a good approach *for things where the alternative is
never to fix it*.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IC2GZWYK5QCBWNFRLSYZE3STEAU3VPBU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin  wrote:
> If we would deprecate using Py_REFCNT as l-value in the docs, but wait
> with the conversion until it was *actually* needed, we would not lose
> anything:
> (...)
> ## CPython nogil fork
>
> In CPython, we cannot change structs that are part of the stable ABI --
> such as PyObject.ob_refcnt. IMO, getting rid of the macros that access
> ob_refcnt is a relatively small part of this issue.

The question is if the nogil optimization is relevant for CPython?
It's a trade-off between backward compatibility and performance. It
seems like some core devs are enthusiast by the idea of getting this
work into CPython: count it in ;-) Getting rid of the GIL is an old
dream which became reality: the abiltiy of using all CPU cores at the
same time!

But not all technical issues have to be solved at once. First, the
*API* can be made compatible with nogil: the Python 3.10 Py_RECNT()
change made this function compatible with nogil. Later, the question
of the stable *ABI* can be studied.


> ## GraalVM Python
>
> This PEP is not enough to get rid of wrappers in GraalVM, yet it forces
> users of CPython to adapt.

Currently, it's considered as second class citizen because of the C
API issues. The idea is to put GraalVM Python at the same level as
CPython.

I would like to put all Python implementation in a fair competition.
The competition is great for innovating and should benefit to
everybody!

The PEP 674 is part of this goal. You're right that this PEP alone is
not enough to remove all wrappers: it doesn't solve all problems.

On the other side, a large PEP like PEP 620 cannot be accepted...
because it requires too many changes at once, it's not technically
possible to implement all changes in a single Python. It must be done
incrementally over multiple Python versions. That's why I'm trying to
split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a
better rationale for each incremental change, and better study the
backward compatibility issues of each incompatible change.


> Until disallowing macros as l-values allows concrete improvements in
> CPython, it should be the job of linters.

Do you know existing linters for C extensions? I don't know any :-(


> FWIW, I do encourage alternative implementations to just not support
> l-value macros. There are only few projects doing this, and the fix is
> often (but not always!) easy. This should be a very small part of
> porting something to a different Python implementation (but I could
> definitely be wrong here, please correct me).

First, PyPy tried to only implement a subset of the C API and promote
cffi for incompatible C extensions. This approach failed.

So PyPy decided to be as compatible as possible with CPython for the C
API. Now PyPy basically supports the whole C API and doesn't want to
drop support for the C extensions "abusing" the C API, like using
macros as l-value.

The problem is that supporting all C API "abuses" requires to
basically copy everything from CPython and it's inefficient. I didn't
mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to
avoid converting efficient PyPy objects to inefficient CPython objects
when they are accessed by the C API (PyPy cpyext module).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UFPN4VUV3Y4ZV4TYJAEAUQUPKOB7SLUU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

Extract of the Backward Compatibility section of the PEP:

"This change does not follow the PEP 387 deprecation process. There is
no known way to emit a deprecation warning only when a macro is used
as a l-value, but not when it's used differently (ex: as a r-value)."

Victor

On Tue, Dec 7, 2021 at 4:41 PM Antoine Pitrou  wrote:
>
> On Tue, 7 Dec 2021 15:39:25 +0100
> Petr Viktorin  wrote:
>
> > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum  wrote:
> > >> How about *not* asking for an exception and just following the PEP 387 
> > >> process? Is that really too burdensome?
> > >
> > > The Backward Compatibility section gives an explanation:
> > >
> > > "This change does not follow the PEP 387 deprecation process. There is no
> > > known way to emit a deprecation warning when a macro is used as a
> > > l-value, but not when it's used differently (ex: r-value)."
> > >
> > > Apart of compiler warnings, one way to implement the PEP 387
> > > "deprecation process" would be to announce the change in two "What's
> > > New in Python 3.X?" documents. But I expect that it will not be
> > > efficient. Extract of the Rejected Idea section:
> > >
> > > "(...) only few developers read the documentation, and only a minority
> > > is tracking changes of the Python C API documentation."
> > >
> > > In my experience, even if a DeprecationWarning is emitted at runtime,
> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > > to remove everything that's deprecated?" discussion and complains
> > > about recent removal of deprecated features, like:
> > >
> > > * collections.MutableMapping was deprecated for 7 Python versions
> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > > * the "U" open() flag was deprecated for 10 Python versions
> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > >
> > > For this specific PEP changes, I consider that the number of impacted
> > > projects is low enough to skip a deprecation process: only 4 projects
> > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > impacted, and 12 have already been updated in the meanwhile. I'm
> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > > has been approved by the Steering Council.
> >
> >
> > The current version of the PEP looks nice, but I don't think the
> > rationale is strong enough.
> > I believe we should:
> > - Mark the l-value usage as deprecated in the docs,
> > - And then do nothing until we find an actual case where this issue
> > blocks development (or is actively dangerous for users).
>
> Is there a way to emit a compilation warning when those macros are used
> as l-values? Even if only enabled on some compilers.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YA3RN7Q2NHTRDMWI7Z6VWUZIAJ4JCD2D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

An idea for unmaintained incompatible C extensions is to update the C
code when a C extension is installed, as part of the build process.
For example, replace "Py_TYPE(obj) = new_type" with "Py_SET_TYPE(obj,
new_type)".

Something similar to the old "2to3" option of setuptools. The
upgrade_pythoncapi.py of the pythoncapi_compat project is already a
partial implementation of idea: it can update the C code, but it's not
integrated in tools like setuptools.

It's just an idea, a full implementation should be designed and written.

Victor

On Tue, Dec 7, 2021 at 5:56 PM Joao S. O. Bueno  wrote:
>
> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing with
> what is met in the "real world".
>
> But if this update can be done by a simple search/replace on the C source of 
> projects,
> I can only perceive two scenarios this will affect: well maintained projects,
>  for which it is fixable in minutes, and  stale packages, no longer released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person trying
> the build can't fix it, but can take the error to a proper, or high 
> visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
>
>
>
> On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou  wrote:
>>
>> On Tue, 7 Dec 2021 15:39:25 +0100
>> Petr Viktorin  wrote:
>>
>> > On 30. 11. 21 19:52, Victor Stinner wrote:
>> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum  
>> > > wrote:
>> > >> How about *not* asking for an exception and just following the PEP 387 
>> > >> process? Is that really too burdensome?
>> > >
>> > > The Backward Compatibility section gives an explanation:
>> > >
>> > > "This change does not follow the PEP 387 deprecation process. There is no
>> > > known way to emit a deprecation warning when a macro is used as a
>> > > l-value, but not when it's used differently (ex: r-value)."
>> > >
>> > > Apart of compiler warnings, one way to implement the PEP 387
>> > > "deprecation process" would be to announce the change in two "What's
>> > > New in Python 3.X?" documents. But I expect that it will not be
>> > > efficient. Extract of the Rejected Idea section:
>> > >
>> > > "(...) only few developers read the documentation, and only a minority
>> > > is tracking changes of the Python C API documentation."
>> > >
>> > > In my experience, even if a DeprecationWarning is emitted at runtime,
>> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
>> > > to remove everything that's deprecated?" discussion and complains
>> > > about recent removal of deprecated features, like:
>> > >
>> > > * collections.MutableMapping was deprecated for 7 Python versions
>> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
>> > > removed again in 3.11
>> > > * the "U" open() flag was deprecated for 10 Python versions
>> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
>> > > removed again in 3.11
>> > >
>> > > For this specific PEP changes, I consider that the number of impacted
>> > > projects is low enough to skip a deprecation process: only 4 projects
>> > > are known to be impacted. One year ago (Python 3.10), 16 were
>> > > impacted, and 12 have already been updated in the meanwhile. I'm
>> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
>> > > has been approved by the Steering Council.
>> >
>> >
>> > The current version of the PEP looks nice, but I don't think the
>> > rationale is strong enough.
>> > I believe we should:
>> > - Mark the l-value usage as deprecated in the docs,
>> > - And then do nothing until we find an actual case where this issue
>> > blocks development (or is actively dangerous for users).
>>
>> Is there a way to emit a compilation warning when those macros are used
>> as l-values? Even if only enabled on some compilers.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT3VK6REJ3BEBVGNDGZCIXWFCA/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
__

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Guido van Rossum]

Hi Victor,

I wonder if there's a role for HPy in this context? What if instead of
evolving the stable ABI and the limited API, instead we were to focus on
first-class support for HPy? Surely 5 release cycles would be enough to
completely remove the stable ABI and perhaps even the limited API in favor
of HPy? Or am I misunderstanding the place of HPy in the ecosystem?

--Guido

PS. Eric wrote an analyzer for C code and checked it in under
Tools/c-analyzer. This currently focuses on finding globals, but perhaps it
forms a good starting point for a linter for C extensions?

On Tue, Dec 7, 2021 at 3:53 PM Victor Stinner  wrote:

> On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin  wrote:
> > If we would deprecate using Py_REFCNT as l-value in the docs, but wait
> > with the conversion until it was *actually* needed, we would not lose
> > anything:
> > (...)
> > ## CPython nogil fork
> >
> > In CPython, we cannot change structs that are part of the stable ABI --
> > such as PyObject.ob_refcnt. IMO, getting rid of the macros that access
> > ob_refcnt is a relatively small part of this issue.
>
> The question is if the nogil optimization is relevant for CPython?
> It's a trade-off between backward compatibility and performance. It
> seems like some core devs are enthusiast by the idea of getting this
> work into CPython: count it in ;-) Getting rid of the GIL is an old
> dream which became reality: the abiltiy of using all CPU cores at the
> same time!
>
> But not all technical issues have to be solved at once. First, the
> *API* can be made compatible with nogil: the Python 3.10 Py_RECNT()
> change made this function compatible with nogil. Later, the question
> of the stable *ABI* can be studied.
>
>
> > ## GraalVM Python
> >
> > This PEP is not enough to get rid of wrappers in GraalVM, yet it forces
> > users of CPython to adapt.
>
> Currently, it's considered as second class citizen because of the C
> API issues. The idea is to put GraalVM Python at the same level as
> CPython.
>
> I would like to put all Python implementation in a fair competition.
> The competition is great for innovating and should benefit to
> everybody!
>
> The PEP 674 is part of this goal. You're right that this PEP alone is
> not enough to remove all wrappers: it doesn't solve all problems.
>
> On the other side, a large PEP like PEP 620 cannot be accepted...
> because it requires too many changes at once, it's not technically
> possible to implement all changes in a single Python. It must be done
> incrementally over multiple Python versions. That's why I'm trying to
> split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a
> better rationale for each incremental change, and better study the
> backward compatibility issues of each incompatible change.
>
>
> > Until disallowing macros as l-values allows concrete improvements in
> > CPython, it should be the job of linters.
>
> Do you know existing linters for C extensions? I don't know any :-(
>
>
> > FWIW, I do encourage alternative implementations to just not support
> > l-value macros. There are only few projects doing this, and the fix is
> > often (but not always!) easy. This should be a very small part of
> > porting something to a different Python implementation (but I could
> > definitely be wrong here, please correct me).
>
> First, PyPy tried to only implement a subset of the C API and promote
> cffi for incompatible C extensions. This approach failed.
>
> So PyPy decided to be as compatible as possible with CPython for the C
> API. Now PyPy basically supports the whole C API and doesn't want to
> drop support for the C extensions "abusing" the C API, like using
> macros as l-value.
>
> The problem is that supporting all C API "abuses" requires to
> basically copy everything from CPython and it's inefficient. I didn't
> mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to
> avoid converting efficient PyPy objects to inefficient CPython objects
> when they are accessed by the C API (PyPy cpyext module).
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UFPN4VUV3Y4ZV4TYJAEAUQUPKOB7SLUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Greg Ewing]

On 8/12/21 4:36 am, Antoine Pitrou wrote:

Is there a way to emit a compilation warning when those macros are used
as l-values? Even if only enabled on some compilers.


Maybe the macro could be written so that it expands to something
with a cast around it? I think gcc has an option for warning about
casts used as l-values.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DG6GBX6OFCK4D2G6CGQFW5QX5JXQPAZO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

For me, HPy is the only valid stable API and stable ABI in the long
term which is efficient on any Python implementation. Its design is
very different than the C API: HPy avoids all C API design mistakes,
it doesn't leak any implementation detail.

HPy can already be used today on CPython, even if it's not directly
provided by CPython.

Providing HPy as a first-class citizen in CPython, as already done in
PyPy, would be great to promote HPy! However, HPy evolves quickly and
so needs to be released more frequently than CPython. At least, we
could promote it more in the C API documentation, as we already
promote Cython.

Promoting the HPy usage doesn't solve any issue listed in PEP 620, 670
and 674 since CPython still has to continue supporting the C API. We
will only be fully free to make any change in Python internals without
having to care about breaking the C API once the *LAST* C extensions
using the C API will disappear... Look at Python 2.7 which is still
used in 2021. I bet that C extensions using the C API are not doing to
disappear soon.

For me, the question is:

=> Is it ok to no longer be able to make any change in Python
internals because of the public C API?

The sub-question is:

=> Is it ok to have a slow deprecation process and wait 5 to 10 years
until it will be possible again to evolve the Python internals?

Obviously, my answer is that we must change the C API as soon as
possible to allow again changing Python internals and help other
Python implementations to support the C API.

--

One option for CPython would be to have a native HPy support, and
emulate the legacy C API with something similar to what PyPy does with
its cpyext module. That may make C extensions using the C API slower
and may increase their memory usage.

Victor


On Wed, Dec 8, 2021 at 1:08 AM Guido van Rossum  wrote:
>
> Hi Victor,
>
> I wonder if there's a role for HPy in this context? What if instead of 
> evolving the stable ABI and the limited API, instead we were to focus on 
> first-class support for HPy? Surely 5 release cycles would be enough to 
> completely remove the stable ABI and perhaps even the limited API in favor of 
> HPy? Or am I misunderstanding the place of HPy in the ecosystem?
>
> --Guido
>
> PS. Eric wrote an analyzer for C code and checked it in under 
> Tools/c-analyzer. This currently focuses on finding globals, but perhaps it 
> forms a good starting point for a linter for C extensions?

-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GH7N6PKCM4775TRAPMCKJT4IRNG64KLP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 674: Disallow using macros as l-value

[Victor Stinner]

Are you talking about gcc -Wignored-qualifiers? It seems like such
warning is only emitted where the function is *defined*, not where the
function is *called*. Example:
---
const int f(void) { return 1; }
int main() { return f(); }
---

Output:
---
$ gcc -Wextra y.c -o y
y.c:1:1: warning: type qualifiers ignored on function return type
[-Wignored-qualifiers]
1 | const int f(void) { return 1; }
  | ^
---

I'm only able to get a compile *error* when a macro is used as a
l-value. Example:
---
struct Point { int x; int y; };
#define POINT_X(p) ((int)p.x)
int main()
{
struct Point p = {1, 2};
int x = POINT_X(p); // r-value ok
POINT_X(p) = 1; // l-value ERROR
return 0;
}
---

With "#define POINT_X(p) (p.x)", the macro can be used as l-value and r-value.

Victor

On Wed, Dec 8, 2021 at 1:43 AM Greg Ewing  wrote:
>
> On 8/12/21 4:36 am, Antoine Pitrou wrote:
> > Is there a way to emit a compilation warning when those macros are used
> > as l-values? Even if only enabled on some compilers.
>
> Maybe the macro could be written so that it expands to something
> with a cast around it? I think gcc has an option for warning about
> casts used as l-values.
>
> --
> Greg
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/DG6GBX6OFCK4D2G6CGQFW5QX5JXQPAZO/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SHQJR3S4WI2OWS6M3J3NFPRYZEHJV7ZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] So many PEPs

[Koos Zevenhoven]

Steering Council, g'day, Hawaii?

("copying" BDFL-emeritus, everyone)

I'm sorry you haven't heard from me in a while. Actually, neither have any
of my other "girl friends" heard from me. I was worried that they might be
sad. Then I found out that one of them was at some kind of party at a club
or something, and got in trouble because she couldn't be reached. I learned
from the newspaper and other media that she indeed was really really sad,
because of this reason.

Yesterday was supposed to be a big day for her, as the president was
throwing a party. But something was missing – because of COVID-19, i
believe. So various kinds of things have been going on, about which I
wasn't sure if i could tell you ;)

Anyway, I really needed to wrap something for her yesterday, but another
day has passed. And that is because of *me*.

In Finland, we have this thing "me", but it means "we". Then we also have
"hän", which means "he"/"she" etc. But using "hän" may feel needlessly
formal, so we often instead use "se", which means "it". And that's that.

Lots of these PEPs coming from everywhere and been going on for quite some
time now!
>>> it = iter(filter(f2, zip(filter(f1, peps), fun(other_peps  # ouch!

I had a laptop with me the other day when going for a swim and somehow the
battery had magically drained while I was swimming, although I thought I
didn't swim that long. So I think my mind (or "taka-raivo", meaning back of
the head; or "back rage"; or perhaps "muscle memory") is playing tricks on
me. Or it could be that I just forgot to charge it. Ok, that's a strange
story. There was another story, which is even stranger, but also quite
embarrassing. So I told you this boring one instead. Oh, and someone
mentioned a frog or prog or something and it seemed it was talking about
me, and someTHING reminded me of krog. Maybe I should have rejected PEPs
like this.

Anyway, today I was reminded of an actual *girl friend* from quite some
time ago, whose early PEP i didn't reject. We once went to India. The food
was good but she thought it was hot ;)

Oh, and there's a girl FRIEND of mine from school. She would call me Koppi,
meaning "knock-π", or "election booth" or something. At some point in
school we somehow ended up in a conversation about God. I had my own
scientific ideas about it, which was very strange in the context so I told
her a simplified version. But still, we keep rejecting each other's PEPs ;)

Hey wait, i'm talking about girls all the time? Ok, music maybe?

I had a two-meter high stack of recyclable paper from over the years. The
stack has fallen over and broken my shoe rack, and now i noticed that, for
some reason, the score of an often requested song was right there:

"I love coffee, i love tea, i love the Java Jive and it loves me",
dismissing the cover solo in the key of D.

(memories)
[.]
(saving the discussions)

This reminds me of someone saying "I hope `it` is not very long". In a
different context some years back, I tried to make someone longer, though
obeying the laws of physics. It was so interesting that even the police got
involved :D. However, there was no significant change in length. Just as
already predicted from special relativity.

There was a PEP about "early binding". That's a bit tricky, but seems to be
big in 日本. It's possible that I have taken an early course on such things.
I have a friend that could be into such things. Anyway, I'm sure there are
special interest groups for "binding" ;)

Recently, we drove with my parents to their summer place, which is not that
far. On the way there, he told me mostly jokes, but was also looking at the
difference between Merkel and Scholz (puntended in here!)

PEPs keep coming in from different places. Even coffee machines are playing
tricks on me.

I can't just reject all the PEPs, can I? So I'm in trouble here! (and I
originally wrote this as one of the first sentences in this email)

I don't even know how many PEPs there are, and whether I yet need yet
another one yet. And whether that's not NaN in some language.

Getting more and more random, and pretty long.

[]

Gooder like this? Yes me can! Let's C!
Started moving my feet.
TAU, i just had to dance a bit or two.

Tread safe.

Santa b*ches, shoe size 45!
Need food, wash my clothes.

Kiitos ja anteeksi.

BTW, tosi hieno E! Ah, yes.


Kind of wishing i could just reject all PEPs at once.

But instead I'm still in big trouble! Eeeek!

Help! Please help! Please, please help me deal with the PEPs!

In the name of loves and doves,
Signed, "sealed", am delivered,
Yours sincerely,
Koos

PS. I'm getting tired, but you are still welcome to rejoice Finland's 104
years of independence (itsenäisyys, technically meaning "being as yourself")

PSS. Currently struggling a bit with it myself; I blame the fact that these
were mostly non-standard PEPs ;)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@pyt