[Python-Dev] Re: Make HAMT available to python script

[Christopher Barker]

On Fri, Apr 1, 2022 at 4:06 AM Steve Dower  wrote:

> The main difference is that 'immutables' offers you a stable/versioned
> interface to use it, while the one that's in CPython is an internal
> implementation detail. If one day we find a better design, we can just
> switch to it, while 'immutables' probably can't. If we've exposed as a
> public interface in the core runtime, it's much more complicated.
>

I don't understand the issue here:

If we expose a "frozendict" as built in python object then only the API of
that object needs to remain stable, not the implementation.

And it seems that's an API that is already clearly defined.

+ 1 from me -- just the other day I was wishing it was there.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MSVMBGTH3GIZR2BXXKXKO52BE5NJAZ2R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Guido van Rossum]

By beta 1 things should be stable (modulo bug fixes). But documentation may
lag. If you can’t figure something out by reading the code by all means ask!

On Fri, Apr 1, 2022 at 11:40 Matthieu Dartiailh 
wrote:

> As the maintainer of bytecode (thanks to Victor), I expect that adding
> support for 3.11 will be challenging at least. However I hoped that by
> waiting for the first beta most changes would be at least documented. What
> would be the best channel to reach people that may clarify how things work
> starting with 3.11 ?
>
> Best
>
> Matthieu Dartiailh
>
> On Fri, Apr 1, 2022, 18:34 Mark Shannon  wrote:
>
>> Hi Gabriele,
>>
>> On 01/04/2022 4:50 pm, Gabriele wrote:
>> > Does this mean that this line in the bytecode library is likely to fail
>> with 3.11, with no way to fix it?
>> >
>>
>> You can pass the exception table the same way you pass all the other
>> arguments.
>> The exception table depends on the code, but that is nothing new. The
>> bytecode library already recomputes the consts, names, etc.
>>
>> TBH, calling `types.CodeType` didn't work for earlier versions either.
>> It just sort of worked, some of the time.
>>
>> Cheers,
>> Mark.
>>
>>
>> >
>> https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
>> <
>> https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
>> >
>> >
>> > On Fri, 1 Apr 2022, 10:40 Victor Stinner, > vstin...@python.org>> wrote:
>> >
>> > I created https://bugs.python.org/issue47185 <
>> https://bugs.python.org/issue47185> to discuss this issue:
>> > either recompute automatically co_exceptiontable, or at least
>> document
>> > the change.
>> >
>> > Victor
>> >
>> > On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner > > wrote:
>> >  >
>> >  > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to
>> CPython"
>> >  > private C API to the internal C API")
>> >  >
>> >  > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico > > wrote:
>> >  > >
>> >  > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <
>> vstin...@python.org > wrote:
>> >  > > > In Python, sadly the types.CodeType type also has a public
>> constructor
>> >  > > > and many projects break at each Python release because the
>> API
>> >  > > > changes. Hopefully, it seems like the new CodeType.replace()
>> method
>> >  > > > added to Python 3.8 mitigated the issue. IMO
>> CodeType.replace() is a
>> >  > > > better abstraction and closer to what developers need in
>> practice.
>> >  > >
>> >  > > It certainly has been for me. When I want to do bytecode
>> hackery, I
>> >  > > usually start by creating a function with def/lambda, then
>> construct a
>> >  > > modified function using f.__code__.replace(). It's the easiest
>> way to
>> >  > > ensure that all the little details are correct.
>> >  >
>> >  > Python 3.11 added the concept of "exception table"
>> >  > (code.co_exceptiontable). You have to build this table, otherwise
>> >  > Python can no longer catch exceptions :-)
>> >  >
>> >  > I don't know how to build this exception table. It seems like
>> >  > currently there is no Python function in the stdlib to build this
>> >  > table.
>> >  >
>> >  > Example:
>> >  > ---
>> >  > def f():
>> >  > try:
>> >  > print("raise")
>> >  > raise ValueError
>> >  > except ValueError:
>> >  > print("except")
>> >  > else:
>> >  > print("else")
>> >  > print("exit func")
>> >  >
>> >  > def g(): pass
>> >  >
>> >  > if 1:
>> >  > code = f.__code__
>> >  > g.__code__ = g.__code__.replace(
>> >  > co_code=code.co_code,
>> >  > co_consts=code.co_consts,
>> >  > co_names=code.co_names,
>> >  > co_flags=code.co_flags,
>> >  > co_stacksize=code.co_stacksize)
>> >  > else:
>> >  > g.__code__ = f.__code__  # this code path works on Python
>> 3.11
>> >  >
>> >  > g()
>> >  > ---
>> >  >
>> >  > Output with Python 3.10 (ok):
>> >  > ---
>> >  > raise
>> >  > except
>> >  > exit func
>> >  > ---
>> >  >
>> >  > Output with Python 3.11 (oops):
>> >  > ---
>> >  > raise
>> >  > Traceback (most recent call last):
>> >  >   ...
>> >  > ValueError
>> >  > ---
>> >  >
>> >  > By the way, this change is not documented at all:
>> >  >
>> >  > * https://docs.python.org/dev/library/types.html#types.CodeType
>> 
>> >  > * https://docs.python.org/dev/whatsnew/3.11.html <
>> https://docs.python.org/dev/whatsnew/3.11.html>
>> >  >
>> >  > I understand that these changes 

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Matthieu Dartiailh]

As the maintainer of bytecode (thanks to Victor), I expect that adding
support for 3.11 will be challenging at least. However I hoped that by
waiting for the first beta most changes would be at least documented. What
would be the best channel to reach people that may clarify how things work
starting with 3.11 ?

Best

Matthieu Dartiailh

On Fri, Apr 1, 2022, 18:34 Mark Shannon  wrote:

> Hi Gabriele,
>
> On 01/04/2022 4:50 pm, Gabriele wrote:
> > Does this mean that this line in the bytecode library is likely to fail
> with 3.11, with no way to fix it?
> >
>
> You can pass the exception table the same way you pass all the other
> arguments.
> The exception table depends on the code, but that is nothing new. The
> bytecode library already recomputes the consts, names, etc.
>
> TBH, calling `types.CodeType` didn't work for earlier versions either.
> It just sort of worked, some of the time.
>
> Cheers,
> Mark.
>
>
> >
> https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
> <
> https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
> >
> >
> > On Fri, 1 Apr 2022, 10:40 Victor Stinner,  vstin...@python.org>> wrote:
> >
> > I created https://bugs.python.org/issue47185 <
> https://bugs.python.org/issue47185> to discuss this issue:
> > either recompute automatically co_exceptiontable, or at least
> document
> > the change.
> >
> > Victor
> >
> > On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner  > wrote:
> >  >
> >  > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to
> CPython"
> >  > private C API to the internal C API")
> >  >
> >  > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico  > wrote:
> >  > >
> >  > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner <
> vstin...@python.org > wrote:
> >  > > > In Python, sadly the types.CodeType type also has a public
> constructor
> >  > > > and many projects break at each Python release because the API
> >  > > > changes. Hopefully, it seems like the new CodeType.replace()
> method
> >  > > > added to Python 3.8 mitigated the issue. IMO
> CodeType.replace() is a
> >  > > > better abstraction and closer to what developers need in
> practice.
> >  > >
> >  > > It certainly has been for me. When I want to do bytecode
> hackery, I
> >  > > usually start by creating a function with def/lambda, then
> construct a
> >  > > modified function using f.__code__.replace(). It's the easiest
> way to
> >  > > ensure that all the little details are correct.
> >  >
> >  > Python 3.11 added the concept of "exception table"
> >  > (code.co_exceptiontable). You have to build this table, otherwise
> >  > Python can no longer catch exceptions :-)
> >  >
> >  > I don't know how to build this exception table. It seems like
> >  > currently there is no Python function in the stdlib to build this
> >  > table.
> >  >
> >  > Example:
> >  > ---
> >  > def f():
> >  > try:
> >  > print("raise")
> >  > raise ValueError
> >  > except ValueError:
> >  > print("except")
> >  > else:
> >  > print("else")
> >  > print("exit func")
> >  >
> >  > def g(): pass
> >  >
> >  > if 1:
> >  > code = f.__code__
> >  > g.__code__ = g.__code__.replace(
> >  > co_code=code.co_code,
> >  > co_consts=code.co_consts,
> >  > co_names=code.co_names,
> >  > co_flags=code.co_flags,
> >  > co_stacksize=code.co_stacksize)
> >  > else:
> >  > g.__code__ = f.__code__  # this code path works on Python 3.11
> >  >
> >  > g()
> >  > ---
> >  >
> >  > Output with Python 3.10 (ok):
> >  > ---
> >  > raise
> >  > except
> >  > exit func
> >  > ---
> >  >
> >  > Output with Python 3.11 (oops):
> >  > ---
> >  > raise
> >  > Traceback (most recent call last):
> >  >   ...
> >  > ValueError
> >  > ---
> >  >
> >  > By the way, this change is not documented at all:
> >  >
> >  > * https://docs.python.org/dev/library/types.html#types.CodeType <
> https://docs.python.org/dev/library/types.html#types.CodeType>
> >  > * https://docs.python.org/dev/whatsnew/3.11.html <
> https://docs.python.org/dev/whatsnew/3.11.html>
> >  >
> >  > I understand that these changes come from the "Zero cost exception
> >  > handling" change:
> >  > https://bugs.python.org/issue40222 <
> https://bugs.python.org/issue40222>
> >  >
> >  > Victor
> >  > --
> >  > Night gathers, and now my watch begins. It shall not end until my
> death.
> >
> >
> >
> > --
> > Night gathers, and now my watch begins. It shall not 

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2022-03-25 - 2022-04-01)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7153 ( +8)
  closed 51763 (+61)
  total  58916 (+69)

Open issues with patches: 2902 


Issues opened (49)
==

#46907: Update Windows and MacOS installer to SQLite 3.38.2
https://bugs.python.org/issue46907  reopened by ned.deily

#47122: Fix the table of methods in the collections.abc documentation
https://bugs.python.org/issue47122  opened by maggyero

#47123: ZipFile.writestr should respect SOURCE_DATE_EPOCH
https://bugs.python.org/issue47123  opened by ghost43

#47124: explore hashlib use of the Apple CryptoKit macOS
https://bugs.python.org/issue47124  opened by gregory.p.smith

#47125: Explore hashlib use of the Windows Crypto API NG
https://bugs.python.org/issue47125  opened by gregory.p.smith

#47131: Speedup test_unparse
https://bugs.python.org/issue47131  opened by jkloth

#47132: Move tests from setobject.c to _testcapimodule
https://bugs.python.org/issue47132  opened by arhadthedev

#47133: enhance unittest to show test name and docstring on one line
https://bugs.python.org/issue47133  opened by ethan.furman

#47134: Document the meaning of the number in OverflowError
https://bugs.python.org/issue47134  opened by steven.daprano

#47135: Allow decimal.localcontext to accept keyword arguments to set 
https://bugs.python.org/issue47135  opened by steven.daprano

#47136: Wrong value assigned automatically to the variable __module__ 
https://bugs.python.org/issue47136  opened by Takuo Matsuoka

#47138: Pin Jinja2 to fix docs build
https://bugs.python.org/issue47138  opened by hugovk

#47139: pthread_sigmask needs SIG_BLOCK behaviour explaination
https://bugs.python.org/issue47139  opened by rpurdie

#47140: configure --enable-optimizations with clang *12* fails to dete
https://bugs.python.org/issue47140  opened by ofekshilon

#47141: EmailMessage may lack Mime-Version
https://bugs.python.org/issue47141  opened by Vlastimil.Z??ma

#47142: Document importlib.resources.abc.Traversable
https://bugs.python.org/issue47142  opened by petr.viktorin

#47143: Add types.copy_class() which updates closures
https://bugs.python.org/issue47143  opened by vstinner

#47144: Allow setting __classcell__
https://bugs.python.org/issue47144  opened by douglas-raillard-arm

#47145: Improve graphlib.TopologicalSort by removing the prepare step
https://bugs.python.org/issue47145  opened by larry

#47147: Allow `return yield from`
https://bugs.python.org/issue47147  opened by pxeger

#47149: DatagramHandler doing DNS lookup on every log message
https://bugs.python.org/issue47149  opened by bmerry

#47150: HTTPRedirectHandler fails on POST for 307 and 308
https://bugs.python.org/issue47150  opened by Jairo Llopis

#47152: Reorganize the re module sources
https://bugs.python.org/issue47152  opened by serhiy.storchaka

#47153: __doc__ should generally be writable
https://bugs.python.org/issue47153  opened by pitrou

#47154: -arch  detection in _osx_support generates false positiv
https://bugs.python.org/issue47154  opened by isuruf

#47156: IDLE: make use of extended SyntaxError info.
https://bugs.python.org/issue47156  opened by terry.reedy

#47158: logging.handlers.SysLogHandler doesn't get cleaned up properly
https://bugs.python.org/issue47158  opened by ngie

#47159: multiprocessing.pool.Pool.apply block infinitely when stressed
https://bugs.python.org/issue47159  opened by harsh8398

#47161: pathlib method relative_to doesnt work with // in paths
https://bugs.python.org/issue47161  opened by John15321

#47164: [C API] Add private "CAST" macros to clean up casts in C code
https://bugs.python.org/issue47164  opened by vstinner

#47165: [C API] Test that the Python C API is compatible with C++
https://bugs.python.org/issue47165  opened by vstinner

#47166: Dataclass transform should ignore TypeAlias variables
https://bugs.python.org/issue47166  opened by thomkeh

#47167: Allow overriding future-task compliance check in  asyncio
https://bugs.python.org/issue47167  opened by asvetlov

#47168: Improvements for stable ABI definition files
https://bugs.python.org/issue47168  opened by petr.viktorin

#47169: Stable ABI: Some optional (#ifdef'd) functions aren't handled 
https://bugs.python.org/issue47169  opened by petr.viktorin

#47174: Define behavior of descriptor-typed fields on dataclasses
https://bugs.python.org/issue47174  opened by debonte

#47175: Allow applications to tune the condition that triggers a GIL r
https://bugs.python.org/issue47175  opened by gregory.p.smith

#47176: Interrupt handling for wasm32-emscripten builds without pthrea
https://bugs.python.org/issue47176  opened by hoodmane

#47177: Frames should store next_instr instead of lasti
https://bugs.python.org/issue47177  opened by brandtbucher

#47179: pymalloc should align to max_align_t

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Mark Shannon]

Hi Gabriele,

On 01/04/2022 4:50 pm, Gabriele wrote:

Does this mean that this line in the bytecode library is likely to fail with 
3.11, with no way to fix it?



You can pass the exception table the same way you pass all the other arguments.
The exception table depends on the code, but that is nothing new. The bytecode 
library already recomputes the consts, names, etc.

TBH, calling `types.CodeType` didn't work for earlier versions either.
It just sort of worked, some of the time.

Cheers,
Mark.



https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
 


On Fri, 1 Apr 2022, 10:40 Victor Stinner, mailto:vstin...@python.org>> wrote:

I created https://bugs.python.org/issue47185 
 to discuss this issue:
either recompute automatically co_exceptiontable, or at least document
the change.

Victor

On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner mailto:vstin...@python.org>> wrote:
 >
 > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
 > private C API to the internal C API")
 >
 > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico mailto:ros...@gmail.com>> wrote:
 > >
 > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner mailto:vstin...@python.org>> wrote:
 > > > In Python, sadly the types.CodeType type also has a public 
constructor
 > > > and many projects break at each Python release because the API
 > > > changes. Hopefully, it seems like the new CodeType.replace() method
 > > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
 > > > better abstraction and closer to what developers need in practice.
 > >
 > > It certainly has been for me. When I want to do bytecode hackery, I
 > > usually start by creating a function with def/lambda, then construct a
 > > modified function using f.__code__.replace(). It's the easiest way to
 > > ensure that all the little details are correct.
 >
 > Python 3.11 added the concept of "exception table"
 > (code.co_exceptiontable). You have to build this table, otherwise
 > Python can no longer catch exceptions :-)
 >
 > I don't know how to build this exception table. It seems like
 > currently there is no Python function in the stdlib to build this
 > table.
 >
 > Example:
 > ---
 > def f():
 >     try:
 >         print("raise")
 >         raise ValueError
 >     except ValueError:
 >         print("except")
 >     else:
 >         print("else")
 >     print("exit func")
 >
 > def g(): pass
 >
 > if 1:
 >     code = f.__code__
 >     g.__code__ = g.__code__.replace(
 >         co_code=code.co_code,
 >         co_consts=code.co_consts,
 >         co_names=code.co_names,
 >         co_flags=code.co_flags,
 >         co_stacksize=code.co_stacksize)
 > else:
 >     g.__code__ = f.__code__  # this code path works on Python 3.11
 >
 > g()
 > ---
 >
 > Output with Python 3.10 (ok):
 > ---
 > raise
 > except
 > exit func
 > ---
 >
 > Output with Python 3.11 (oops):
 > ---
 > raise
 > Traceback (most recent call last):
 >   ...
 > ValueError
 > ---
 >
 > By the way, this change is not documented at all:
 >
 > * https://docs.python.org/dev/library/types.html#types.CodeType 

 > * https://docs.python.org/dev/whatsnew/3.11.html 

 >
 > I understand that these changes come from the "Zero cost exception
 > handling" change:
 > https://bugs.python.org/issue40222 
 >
 > Victor
 > --
 > Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.

___
Python-Dev mailing list -- python-dev@python.org 

To unsubscribe send an email to python-dev-le...@python.org 

https://mail.python.org/mailman3/lists/python-dev.python.org/ 

Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
 

Code of Conduct: http://python.org/psf/codeofconduct/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to 

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Guido van Rossum]

On Fri, Apr 1, 2022 at 8:56 AM Gabriele  wrote:

> Does this mean that this line in the bytecode library is likely to fail
> with 3.11, with no way to fix it?
>
>
> https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
>

Yes, that constructor is not considered stable.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SZ6CASDWZY3AXQBF6VFYHFBBFWIY2CNI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Gabriele]

Does this mean that this line in the bytecode library is likely to fail
with 3.11, with no way to fix it?

https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398

On Fri, 1 Apr 2022, 10:40 Victor Stinner,  wrote:

> I created https://bugs.python.org/issue47185 to discuss this issue:
> either recompute automatically co_exceptiontable, or at least document
> the change.
>
> Victor
>
> On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner 
> wrote:
> >
> > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
> > private C API to the internal C API")
> >
> > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico  wrote:
> > >
> > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner 
> wrote:
> > > > In Python, sadly the types.CodeType type also has a public
> constructor
> > > > and many projects break at each Python release because the API
> > > > changes. Hopefully, it seems like the new CodeType.replace() method
> > > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > > > better abstraction and closer to what developers need in practice.
> > >
> > > It certainly has been for me. When I want to do bytecode hackery, I
> > > usually start by creating a function with def/lambda, then construct a
> > > modified function using f.__code__.replace(). It's the easiest way to
> > > ensure that all the little details are correct.
> >
> > Python 3.11 added the concept of "exception table"
> > (code.co_exceptiontable). You have to build this table, otherwise
> > Python can no longer catch exceptions :-)
> >
> > I don't know how to build this exception table. It seems like
> > currently there is no Python function in the stdlib to build this
> > table.
> >
> > Example:
> > ---
> > def f():
> > try:
> > print("raise")
> > raise ValueError
> > except ValueError:
> > print("except")
> > else:
> > print("else")
> > print("exit func")
> >
> > def g(): pass
> >
> > if 1:
> > code = f.__code__
> > g.__code__ = g.__code__.replace(
> > co_code=code.co_code,
> > co_consts=code.co_consts,
> > co_names=code.co_names,
> > co_flags=code.co_flags,
> > co_stacksize=code.co_stacksize)
> > else:
> > g.__code__ = f.__code__  # this code path works on Python 3.11
> >
> > g()
> > ---
> >
> > Output with Python 3.10 (ok):
> > ---
> > raise
> > except
> > exit func
> > ---
> >
> > Output with Python 3.11 (oops):
> > ---
> > raise
> > Traceback (most recent call last):
> >   ...
> > ValueError
> > ---
> >
> > By the way, this change is not documented at all:
> >
> > * https://docs.python.org/dev/library/types.html#types.CodeType
> > * https://docs.python.org/dev/whatsnew/3.11.html
> >
> > I understand that these changes come from the "Zero cost exception
> > handling" change:
> > https://bugs.python.org/issue40222
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ARKTYXZMYEW3JHNTR7RNLIDDIIOMU7B6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Mark Shannon]

Hi,

On 22/03/2022 6:07 pm, Victor Stinner wrote:

Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054


I just want to say that it is important the API is easy to use and access.

Otherwise there is a temptation to directly set the underlying function 
pointer, and that means we don't get to see when PEP 523 is used.
We might then mis-optimize on the assumption that PEP 523 isn't used.

Cheers,
Mark.



API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L2E2HLUT757ROREKWIAD7M2HIHNUJ3VE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[zhang kai]

Hi, Steve. Thanks for your detailed explanation. Indeed I already saw the
API discussion in PEP 603. It's much easier to make the decision in a
third-party library. I think we will be fine with the immutables library.

On Fri, Apr 1, 2022 at 7:05 PM Steve Dower  wrote:

> On 4/1/2022 11:48 AM, zhang kai wrote:
> > Thanks Victor and Pablo. I will check the discussion of PEP 603. It's a
> > little weird to use the immutables library when it's code in already in
> > CPython but I'm glad it's an option.
>
> The main difference is that 'immutables' offers you a stable/versioned
> interface to use it, while the one that's in CPython is an internal
> implementation detail. If one day we find a better design, we can just
> switch to it, while 'immutables' probably can't. If we've exposed as a
> public interface in the core runtime, it's much more complicated.
>
> (For what it's worth, the major thing that held up contextvars in the
> first place was justifying why it needed a new data structure that
> wasn't already in the core runtime. So we didn't adopt it lightly, and
> making sure we kept the freedom to change it was an important compromise.)
>
> There are plenty of other things in this same category, and because we
> want to keep things as stable as possible while also improving
> performance and reliability, we have to keep pretty tight limits on what
> we promise will remain stable. Most of our discussions are about finding
> this balance ;)
>
> Cheers,
> Steve
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4OZ2DNZ3Q3LY6CUGJEUTXGZRNIX73FYQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 01. 04. 22 11:01, Victor Stinner wrote:

Hi,

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850

The following APIs have been moved to the internal C API:

- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()


Really?
I haven't seen any support for that in this thread or on the issue, 
except from you.
Now, the people who'd like a non-breaking solution will now need rush to 
develop and merge one until the next release, or the API breaks for 
users and it'll be too late to do anything about it. Meanwhile, you're 
off to make more changes. That is, frankly, very frustrating.


Could you please stop unilaterally breaking documented API?



If you use any of these API in your debugger/profiler project, you
have do add something like the code below to your project:
---
#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#if PY_VERSION_HEX >= 0x030B00A7
#  include  // _PyInterpreterState_SetEvalFrameFunc()
#  include   // _PyEval_EvalFrameDefault()
#endif
---


IMO, this is a terrible suggestion that undermines the whole point of 
having a private API.




Contact me if you need help to update your affected projects.

IMO PEP 523 doesn't have to be updated since it already says that the
APIs are private.

Since these APIs were added by PEP 523, I documented these changes in
What's New in Python 3.11 > C API > Porting to Python 3.11,even if
these APIs are private.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGBYB2WKOUZUA6LXNLMMLY7PKEQG3FZW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Steve Dower]

On 4/1/2022 11:48 AM, zhang kai wrote:
Thanks Victor and Pablo. I will check the discussion of PEP 603. It's a 
little weird to use the immutables library when it's code in already in 
CPython but I'm glad it's an option.


The main difference is that 'immutables' offers you a stable/versioned 
interface to use it, while the one that's in CPython is an internal 
implementation detail. If one day we find a better design, we can just 
switch to it, while 'immutables' probably can't. If we've exposed as a 
public interface in the core runtime, it's much more complicated.


(For what it's worth, the major thing that held up contextvars in the 
first place was justifying why it needed a new data structure that 
wasn't already in the core runtime. So we didn't adopt it lightly, and 
making sure we kept the freedom to change it was an important compromise.)


There are plenty of other things in this same category, and because we 
want to keep things as stable as possible while also improving 
performance and reliability, we have to keep pretty tight limits on what 
we promise will remain stable. Most of our discussions are about finding 
this balance ;)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KDJLDBZ5FIMUCVBKGEWCA2T4NGTZB4CW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[zhang kai]

Thanks Victor and Pablo. I will check the discussion of PEP 603. It's a
little weird to use the immutables library when it's code in already in
CPython but I'm glad it's an option.

Kai

On Fri, Apr 1, 2022 at 6:14 PM Pablo Galindo Salgado 
wrote:

> You may want to check PEP 603, which more or less proposes this (the
> author of the pep is the author of the HAMT code)
>  check https://peps.python.org/pep-0603/
>
> Alternatively, there is already a pypi package with this code:
>
> https://pypi.org/project/immutables/
>
> Regards from cloudy London,
> Pablo
>
> On Fri, 1 Apr 2022, 10:34 zhang kai,  wrote:
>
>> Hi,
>>
>> HAMT is a very useful immutable mapping type. Currently CPython use it
>> internally to implement contextvar. Considering immutable data structure is
>> very useful I hope we can make it available to python script(maybe via
>> collections module).
>>
>> Immutable data structures are fundamental parts of our project. Currently
>> we use a full-featured python immutable data library called pyrsistent.
>> Pyrsistent is very powerful, however the map type in it is implemented in
>> python script not in C. It becomes slow when the data set is relatively
>> large.
>>
>> On the other hand, CPython now already has an immutable mapping type in
>> it. I think maybe it’s a good idea to make it public? Many projects can
>> benefit from it I believe.
>>
>> Here is a talk given by the author of javascript immutable-js library
>> explain why immutable data structures are powerful:
>> https://www.youtube.com/watch?v=I7IdS-PbEgI
>>
>> Pyristent: https://github.com/tobgu/pyrsistent
>>
>> What do you think?
>>
>> Cheers,
>> Kai
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/2WYPX44WBFS2ZMZFNMDFMRPROB7G34JQ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RAD37ETHP5BXR4QZQKZIBVLHPEGK2HYO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 4/1/2022 10:01 AM, Victor Stinner wrote:

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850


So what was the point of this discussion then?

I don't see any additional discussion on the bug, and the prevailing 
opinion from actual users of this API is that it probably shouldn't 
change, and certainly shouldn't become internal without additional 
guarantees about stability.


Did we all just waste a whole lot of electrons discussing a foregone 
conclusion?


(My apologies to the people I invited into this thread. I genuinely 
thought it would help to have outside perspective on this potential change.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZGWG3KKGGWI7HLYSJUDQVTXFUNGZKD3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Pablo Galindo Salgado]

You may want to check PEP 603, which more or less proposes this (the author
of the pep is the author of the HAMT code)
 check https://peps.python.org/pep-0603/

Alternatively, there is already a pypi package with this code:

https://pypi.org/project/immutables/

Regards from cloudy London,
Pablo

On Fri, 1 Apr 2022, 10:34 zhang kai,  wrote:

> Hi,
>
> HAMT is a very useful immutable mapping type. Currently CPython use it
> internally to implement contextvar. Considering immutable data structure is
> very useful I hope we can make it available to python script(maybe via
> collections module).
>
> Immutable data structures are fundamental parts of our project. Currently
> we use a full-featured python immutable data library called pyrsistent.
> Pyrsistent is very powerful, however the map type in it is implemented in
> python script not in C. It becomes slow when the data set is relatively
> large.
>
> On the other hand, CPython now already has an immutable mapping type in
> it. I think maybe it’s a good idea to make it public? Many projects can
> benefit from it I believe.
>
> Here is a talk given by the author of javascript immutable-js library
> explain why immutable data structures are powerful:
> https://www.youtube.com/watch?v=I7IdS-PbEgI
>
> Pyristent: https://github.com/tobgu/pyrsistent
>
> What do you think?
>
> Cheers,
> Kai
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2WYPX44WBFS2ZMZFNMDFMRPROB7G34JQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WSNGWKINHMVM2BWOO545FYEOSIXTZKOW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Make HAMT available to python script

[Victor Stinner]

Hi,

In 2019, Yury Selivanov, who added HAMT and contextvars to Python,
wrote PEP 603 "Adding a frozenmap type to collections":
https://peps.python.org/pep-0603/

Sadly, the PEP was stuck in discussions:

* 
https://discuss.python.org/t/pep-603-adding-a-frozenmap-type-to-collections/2318
* https://discuss.python.org/t/pep-603-frozenmap-vs-my-frozendict/2473

I don't think that differences between dict and frozenmap was a
blocker issue. In fact, if I recall correctly, there was no blocker
issue.

Anyway, Yury is also maintaining the "immutables" 3rd party module
which implements frozenmap:
https://pypi.org/project/immutables/

Victor

On Fri, Apr 1, 2022 at 11:37 AM zhang kai  wrote:
>
> Hi,
>
> HAMT is a very useful immutable mapping type. Currently CPython use it 
> internally to implement contextvar. Considering immutable data structure is 
> very useful I hope we can make it available to python script(maybe via 
> collections module).
>
> Immutable data structures are fundamental parts of our project. Currently we 
> use a full-featured python immutable data library called pyrsistent. 
> Pyrsistent is very powerful, however the map type in it is implemented in 
> python script not in C. It becomes slow when the data set is relatively large.
>
> On the other hand, CPython now already has an immutable mapping type in it. I 
> think maybe it’s a good idea to make it public? Many projects can benefit 
> from it I believe.
>
> Here is a talk given by the author of javascript immutable-js library explain 
> why immutable data structures are powerful: 
> https://www.youtube.com/watch?v=I7IdS-PbEgI
>
> Pyristent: https://github.com/tobgu/pyrsistent
>
> What do you think?
>
> Cheers,
> Kai
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/2WYPX44WBFS2ZMZFNMDFMRPROB7G34JQ/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PQ42TTASV3ZOBJUTDHARTVTPY5EPIW57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: code.replace() and Python 3.11 exception table

[Victor Stinner]

I created https://bugs.python.org/issue47185 to discuss this issue:
either recompute automatically co_exceptiontable, or at least document
the change.

Victor

On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner  wrote:
>
> ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
> private C API to the internal C API")
>
> On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico  wrote:
> >
> > On Fri, 1 Apr 2022 at 19:51, Victor Stinner  wrote:
> > > In Python, sadly the types.CodeType type also has a public constructor
> > > and many projects break at each Python release because the API
> > > changes. Hopefully, it seems like the new CodeType.replace() method
> > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > > better abstraction and closer to what developers need in practice.
> >
> > It certainly has been for me. When I want to do bytecode hackery, I
> > usually start by creating a function with def/lambda, then construct a
> > modified function using f.__code__.replace(). It's the easiest way to
> > ensure that all the little details are correct.
>
> Python 3.11 added the concept of "exception table"
> (code.co_exceptiontable). You have to build this table, otherwise
> Python can no longer catch exceptions :-)
>
> I don't know how to build this exception table. It seems like
> currently there is no Python function in the stdlib to build this
> table.
>
> Example:
> ---
> def f():
> try:
> print("raise")
> raise ValueError
> except ValueError:
> print("except")
> else:
> print("else")
> print("exit func")
>
> def g(): pass
>
> if 1:
> code = f.__code__
> g.__code__ = g.__code__.replace(
> co_code=code.co_code,
> co_consts=code.co_consts,
> co_names=code.co_names,
> co_flags=code.co_flags,
> co_stacksize=code.co_stacksize)
> else:
> g.__code__ = f.__code__  # this code path works on Python 3.11
>
> g()
> ---
>
> Output with Python 3.10 (ok):
> ---
> raise
> except
> exit func
> ---
>
> Output with Python 3.11 (oops):
> ---
> raise
> Traceback (most recent call last):
>   ...
> ValueError
> ---
>
> By the way, this change is not documented at all:
>
> * https://docs.python.org/dev/library/types.html#types.CodeType
> * https://docs.python.org/dev/whatsnew/3.11.html
>
> I understand that these changes come from the "Zero cost exception
> handling" change:
> https://bugs.python.org/issue40222
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Make HAMT available to python script

[zhang kai]

Hi,

HAMT is a very useful immutable mapping type. Currently CPython use it 
internally to implement contextvar. Considering immutable data structure is 
very useful I hope we can make it available to python script(maybe via 
collections module).

Immutable data structures are fundamental parts of our project. Currently we 
use a full-featured python immutable data library called pyrsistent. Pyrsistent 
is very powerful, however the map type in it is implemented in python script 
not in C. It becomes slow when the data set is relatively large.

On the other hand, CPython now already has an immutable mapping type in it. I 
think maybe it’s a good idea to make it public? Many projects can benefit from 
it I believe.

Here is a talk given by the author of javascript immutable-js library explain 
why immutable data structures are powerful: 
https://www.youtube.com/watch?v=I7IdS-PbEgI

Pyristent: https://github.com/tobgu/pyrsistent

What do you think?

Cheers,
Kai
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2WYPX44WBFS2ZMZFNMDFMRPROB7G34JQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] code.replace() and Python 3.11 exception table

[Victor Stinner]

("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
private C API to the internal C API")

On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico  wrote:
>
> On Fri, 1 Apr 2022 at 19:51, Victor Stinner  wrote:
> > In Python, sadly the types.CodeType type also has a public constructor
> > and many projects break at each Python release because the API
> > changes. Hopefully, it seems like the new CodeType.replace() method
> > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > better abstraction and closer to what developers need in practice.
>
> It certainly has been for me. When I want to do bytecode hackery, I
> usually start by creating a function with def/lambda, then construct a
> modified function using f.__code__.replace(). It's the easiest way to
> ensure that all the little details are correct.

Python 3.11 added the concept of "exception table"
(code.co_exceptiontable). You have to build this table, otherwise
Python can no longer catch exceptions :-)

I don't know how to build this exception table. It seems like
currently there is no Python function in the stdlib to build this
table.

Example:
---
def f():
try:
print("raise")
raise ValueError
except ValueError:
print("except")
else:
print("else")
print("exit func")

def g(): pass

if 1:
code = f.__code__
g.__code__ = g.__code__.replace(
co_code=code.co_code,
co_consts=code.co_consts,
co_names=code.co_names,
co_flags=code.co_flags,
co_stacksize=code.co_stacksize)
else:
g.__code__ = f.__code__  # this code path works on Python 3.11

g()
---

Output with Python 3.10 (ok):
---
raise
except
exit func
---

Output with Python 3.11 (oops):
---
raise
Traceback (most recent call last):
  ...
ValueError
---

By the way, this change is not documented at all:

* https://docs.python.org/dev/library/types.html#types.CodeType
* https://docs.python.org/dev/whatsnew/3.11.html

I understand that these changes come from the "Zero cost exception
handling" change:
https://bugs.python.org/issue40222

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KWSPCLXDHBAP2U4LBSMLQEOC7LREDMPB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

Hi,

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850

The following APIs have been moved to the internal C API:

- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()

If you use any of these API in your debugger/profiler project, you
have do add something like the code below to your project:
---
#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#if PY_VERSION_HEX >= 0x030B00A7
#  include  // _PyInterpreterState_SetEvalFrameFunc()
#  include   // _PyEval_EvalFrameDefault()
#endif
---

Contact me if you need help to update your affected projects.

IMO PEP 523 doesn't have to be updated since it already says that the
APIs are private.

Since these APIs were added by PEP 523, I documented these changes in
What's New in Python 3.11 > C API > Porting to Python 3.11,even if
these APIs are private.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DNJC6U36CDA7S7ATEGAMUPABBSEIYHC4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Chris Angelico]

On Fri, 1 Apr 2022 at 19:51, Victor Stinner  wrote:
> In Python, sadly the types.CodeType type also has a public constructor
> and many projects break at each Python release because the API
> changes. Hopefully, it seems like the new CodeType.replace() method
> added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> better abstraction and closer to what developers need in practice.

It certainly has been for me. When I want to do bytecode hackery, I
usually start by creating a function with def/lambda, then construct a
modified function using f.__code__.replace(). It's the easiest way to
ensure that all the little details are correct.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JJO6LDWA4BE7SFTEF2VP6NLSIX6ZBCGD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum  wrote:
> In the not so distant past I have proposed to introduce a new category, 
> "Unstable APIs". These are public but are not guaranteed to be backwards 
> compatible in feature releases (though I feel they should remain so in bugfix 
> releases).
>
> I'm not sure whether those should have a leading underscore or not. Perhaps 
> (like some other languages do and like maybe we've used in a few places) the 
> name could just include the word "Unstable"?

I recall discussions about PyCode_New(). IMO this API should not be
public at all. It leaks way too many implementation details: cell
variables, optimization for bytecode offset to line and column
numbers, exception table, etc. This API changed often and will
continue to change.

It's not the right abstraction level. We just exposed the function
because it was technically possible and it was convenient since Python
consumes its own C API. The internal C API was created to draw a line
between what API can be consumed outside Python (public) and what API
must not be used outside Python (internal) unless you're writing a
debugger or other uncommon very specific use case. The main difference
is the warranties provided (public) or not (internal) by Python:
tests, documentation, backward compatibility.

In Python, sadly the types.CodeType type also has a public constructor
and many projects break at each Python release because the API
changes. Hopefully, it seems like the new CodeType.replace() method
added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
better abstraction and closer to what developers need in practice.

I'm not convinced that advertising an API as being Unstable (in the
documentation?) is going to solve any kind of problem. People love to
use private APIs, and they do it simply because it's technically
possible :-) At the end of the day, we have to help them updating
their code, otherwise we (at least my Red Hat team) cannot update
Python.

I designed the internal C API to be more annoying to be used (need to
define a macro, need more specific #include) in the hope that people
will think twice before using it :-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin  wrote:
> Perhaps we need a new "tier" of C API for debuggers -- API that's
> guaranteed stable for a major release, and if it's changed it should
> always break with compile errors (e.g. the function gets a new
> argument), rather than silently change semantics.
> The internal API Cython & greenlet need might go it this category too.

We should provide public C API functions for Cython and greenlet
needs: well tested and documented functions. There is an on-going work
in Python 3.11:

* https://docs.python.org/dev/c-api/frame.html
* https://github.com/faster-cpython/ideas/issues/309
* https://bugs.python.org/issue40421

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H64KULVN5O4MXPWWYMO2VBJRTLNNWBYX/
Code of Conduct: http://python.org/psf/codeofconduct/