[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Tue, Sep 7, 2021 at 10:00 AM Stefan Behnel  wrote:

> Guido van Rossum schrieb am 07.09.21 um 00:44:
> > In addition, I just heard from the SC that they've approved the
> exception.
> > So we will remove these two APIs from 3.11 without deprecation.
>


> Erm, hang on – when I wrote that I'm fine with *changing* them, I wasn't
> thinking of actually *removing* them. At least not both. PyCode_NewEmpty()
> isn't a good replacement since it takes low level arguments … char*
> instead
> of Python strings. It's good for the simple use case that it was written
> for (and Cython already uses it for that), but not so great for anything
> beyond that.
>

Is the issue that you want to specify a few additional simple arguments, or
that you already have unicode objects and you don't want to have them
converted back to char * (which will then be converted to new unicode
objects by PyCode_NewEmpty)?


> What I could try is to create only a single dummy code object and then
> always call .replace() on it to create new ones. But that seems hackish
> and
> requires managing yet another bit of global state across static and
> generated code parts.
>

I can't argue with the need for an extra bit of global state, but to me,
calling .replace() is not hackish, it's the best way to create new code
objects -- it works for older Python versions and will keep working in the
future.


> I could also switch to _PyCode_New(), though it's not exactly what I would
> call an attractive option, both for usability reasons and its future API
> stability. (Cython also still generates C89 code, i.e. no partial struct
> initialisations.)
>

That's too bad. With the new struct initializations it's actually more
usable and more stable going forward.


> Any suggestions?
>
> Stefan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/B6WFHGVAASF4MFUMPIHBZEUCPXVANU7D/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AZU3VG26JMU7W3WE6G56XOAZH3Y6ETXI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Stefan Behnel]

Guido van Rossum schrieb am 07.09.21 um 00:44:

In addition, I just heard from the SC that they've approved the exception.
So we will remove these two APIs from 3.11 without deprecation.
Erm, hang on – when I wrote that I'm fine with *changing* them, I wasn't 
thinking of actually *removing* them. At least not both. PyCode_NewEmpty() 
isn't a good replacement since it takes low level arguments … char* instead 
of Python strings. It's good for the simple use case that it was written 
for (and Cython already uses it for that), but not so great for anything 
beyond that.


What I could try is to create only a single dummy code object and then 
always call .replace() on it to create new ones. But that seems hackish and 
requires managing yet another bit of global state across static and 
generated code parts.


I could also switch to _PyCode_New(), though it's not exactly what I would 
call an attractive option, both for usability reasons and its future API 
stability. (Cython also still generates C89 code, i.e. no partial struct 
initialisations.)


Any suggestions?

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B6WFHGVAASF4MFUMPIHBZEUCPXVANU7D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

Oh, I didn't know this *existing* C API function:

PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno)

So Cython could be modified to use it, no?

Victor

On Tue, Sep 7, 2021 at 12:44 AM Guido van Rossum  wrote:
>
> On Fri, Sep 3, 2021 at 4:12 PM Victor Stinner  wrote:
>>
>> On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum  wrote:
>> > FWIW I've applied for an exception from the two-release deprecation policy 
>> > from the SC:
>> > https://github.com/python/steering-council/issues/75
>>
>> On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I
>> didn't check how many are using Cython.
>
>
> Most of them. :-)
>
> I wrote a script that to do a similar search on the 4000 most popular 
> packages, disregarding Cython-generated files (these have "/* Generated by 
> Cython  */" in their first line). Now the list collapsed to this:
>
> Cython-3.0a7.tar.gz: 11 hits in 3 files
> frozendict-2.0.6.tar.gz: 14 hits in 8 files
> gevent-21.8.0.tar.gz: 1 hits in 1 files
> JPype1-1.3.0.tar.gz: 1 hits in 1 files
> mypy-0.910.tar.gz: 2 hits in 1 files
> reportlab-3.6.1.tar.gz: 1 hits in 1 files
> setuptools-9.1.tar.gz: 1 hits in 1 files
>
> Of these:
>
> Cython: obviously :-)
> frozendict: calls PyCode_NewEmpty; seems to include modified CPython headers
> gevent: Uses Cython's __Pyx_PyCode_New in a generated .h file
> JPype: calls PyCode_NewEmpty
> mypy: PyCode_NewEmpty mentioned in a comment
> reportlab: calls PyCode_NewEmpty
> setuptools: in a file generated by Pyrex (Cython's predecessor)
>
> There wasn't a single call to PyCode_NewWithPosOnlyArgs in any of these apart 
> from Cython.
>
> In addition, I just heard from the SC that they've approved the exception. So 
> we will remove these two APIs from 3.11 without deprecation. I've filed 
> https://bugs.python.org/issue45122 to get this done (looking for volunteers).
>
> --
> --Guido van Rossum (python.org/~guido)
> Pronouns: he/him (why is my pronoun here?)



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VQOKM6V7UJA2M6LBZ4JPHLVRK3OCHNRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Fri, Sep 3, 2021 at 4:12 PM Victor Stinner  wrote:

> On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum  wrote:
> > FWIW I've applied for an exception from the two-release deprecation
> policy from the SC:
> > https://github.com/python/steering-council/issues/75
>
> On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I
> didn't check how many are using Cython.
>

Most of them. :-)

I wrote a script that to do a similar search on the 4000 most popular
packages, disregarding Cython-generated files (these have "/* Generated by
Cython  */" in their first line). Now the list collapsed to this:

Cython-3.0a7.tar.gz: 11 hits in 3 files
frozendict-2.0.6.tar.gz: 14 hits in 8 files
gevent-21.8.0.tar.gz: 1 hits in 1 files
JPype1-1.3.0.tar.gz: 1 hits in 1 files
mypy-0.910.tar.gz: 2 hits in 1 files
reportlab-3.6.1.tar.gz: 1 hits in 1 files
setuptools-9.1.tar.gz: 1 hits in 1 files

Of these:

Cython: obviously :-)
frozendict: calls PyCode_NewEmpty; seems to include modified CPython headers
gevent: Uses Cython's __Pyx_PyCode_New in a generated .h file
JPype: calls PyCode_NewEmpty
mypy: PyCode_NewEmpty mentioned in a comment
reportlab: calls PyCode_NewEmpty
setuptools: in a file generated by Pyrex (Cython's predecessor)

There wasn't a single call to PyCode_NewWithPosOnlyArgs in any of these
apart from Cython.

In addition, I just heard from the SC that they've approved the exception.
So we will remove these two APIs from 3.11 without deprecation. I've filed
https://bugs.python.org/issue45122 to get this done (looking for
volunteers).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ABU7TDQUOGKDQDK5WAZGUBYO2BDIYUND/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum  wrote:
> FWIW I've applied for an exception from the two-release deprecation policy 
> from the SC:
> https://github.com/python/steering-council/issues/75

On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I
didn't check how many are using Cython.

yarl-1.6.3.tar.gz
wsaccel-0.6.3.tar.gz
wordcloud-1.8.1.tar.gz
uvloop-0.16.0.tar.gz
tslearn-0.5.2.tar.gz
uamqp-1.4.1.tar.gz
tweedledum-1.1.0.tar.gz
tinycss-0.4.tar.gz
thriftpy-0.3.9.tar.gz
thriftpy2-0.4.14.tar.gz
Theano-PyMC-1.1.2.tar.gz
Theano-1.0.5.tar.gz
TA-Lib-0.4.21.tar.gz
tables-3.6.1.tar.gz
ssh2-python-0.26.0.tar.gz
srsly-2.4.1.tar.gz
ssh-python-0.9.0.tar.gz
statsmodels-0.12.2.tar.gz
sphinx-gallery-0.9.0.tar.gz
sktime-0.7.0.tar.gz
Shapely-1.7.1.tar.gz
wxPython-4.1.1.tar.gz
scikit-image-0.18.2.tar.gz
sasl-0.3.1.tar.gz
scikit-surprise-1.1.1.tar.gz
s2sphere-0.2.5.tar.gz
ruptures-1.1.4.tar.gz
runstats-2.0.0.tar.gz
ruamel.yaml.clib-0.2.6.tar.gz
reedsolo-1.5.4.tar.gz
recordclass-0.15.1.tar.gz
reportlab-3.6.1.tar.gz
rasterio-1.2.6.tar.gz
rapidfuzz-1.4.1.tar.gz
qiskit-terra-0.18.1.tar.gz
pyzmq-22.2.1.tar.gz
pyxDamerauLevenshtein-1.7.0.tar.gz
PyWavelets-1.1.1.tar.gz
python-crfsuite-0.9.7.tar.gz
py_spy-0.3.8.tar.gz
pysimdjson-4.0.2.tar.gz
pysam-0.16.0.1.tar.gz
pypcap-1.2.3.tar.gz
pyngrok-5.0.6.tar.gz
PyLBFGS-0.2.0.13.tar.gz
pyjq-2.5.2.tar.gz
pyhacrf-datamade-0.2.5.tar.gz
pyflux-0.4.15.tar.gz
pygame-2.0.1.tar.gz
pyemd-0.5.1.tar.gz
pydevd-2.4.1.tar.gz
pydevd-pycharm-212.5080.18.tar.gz
plyvel-1.3.0.tar.gz
pmdarima-1.8.2.tar.gz
peewee-3.14.4.tar.gz
paramiko-2.7.2.tar.gz
osmium-3.2.0.tar.gz
orderedset-2.0.3.tar.gz
numpydoc-1.1.0.tar.gz
numdifftools-0.9.40.tar.gz
numba-0.53.1.tar.gz
numcodecs-0.8.1.tar.gz
NetfilterQueue-0.8.1.tar.gz
neobolt-1.7.17.tar.gz
Naked-0.1.31.tar.gz
mypy-0.910.tar.gz
msgpack-python-0.5.6.tar.gz
msgpack-1.0.2.tar.gz
mojimoji-0.0.11.tar.gz
mpi4py-3.1.1.tar.gz
matrixprofile-1.1.10.tar.gz
marisa-trie-0.7.7.tar.gz
lupa-1.9.tar.gz
lxml-4.6.3.tar.gz
lsm-db-0.6.4.tar.gz
linearmodels-4.24.tar.gz
lightfm-1.16.tar.gz
Levenshtein-0.13.0.tar.gz
leven-1.0.4.tar.gz
lda-2.0.0.tar.gz
jsonobject-0.9.10.tar.gz
jq-1.2.1.tar.gz
JPype1-1.3.0.tar.gz
jenkspy-0.2.0.tar.gz
implicit-0.4.4.tar.gz
imgui-1.3.0.tar.gz
imbalanced-learn-0.8.0.tar.gz
imagecodecs-2021.7.30.tar.gz
httptools-0.3.0.tar.gz
httpretty-1.1.4.tar.gz
hmmlearn-0.2.6.tar.gz
hdbscan-0.8.27.tar.gz
gssapi-1.6.14.tar.gz
grpcio-tools-1.39.0.tar.gz
grpcio-1.39.0.tar.gz
graphene-federation-0.1.0.tar.gz
GPy-1.10.0.tar.gz
gluonnlp-0.10.0.tar.gz
gevent-21.8.0.tar.gz
gensim-4.0.1.tar.gz
fuzzyset-0.0.19.tar.gz
fuzzysearch-0.7.3.tar.gz
frozendict-2.0.6.tar.gz
flower-1.0.0.tar.gz
Fiona-1.8.20.tar.gz
fastrlock-0.6.tar.gz
fastparquet-0.7.1.tar.gz
fastdtw-0.3.4.tar.gz
fastavro-1.4.4.tar.gz
edlib-1.3.8.post2.tar.gz
editdistance-0.5.3.tar.gz
econml-0.12.0.tar.gz
dtaidistance-2.3.2.tar.gz
DoubleMetaphone-0.1.tar.gz
django-localflavor-3.1.tar.gz
dependency-injector-4.35.2.tar.gz
dedupe-hcluster-0.3.8.tar.gz
dedupe-2.0.8.tar.gz
ddtrace-0.51.2.tar.gz
cytoolz-0.11.0.tar.gz
Cython-0.29.24.tar.gz
correctionlib-2.0.0.tar.gz
clickhouse-driver-0.2.1.tar.gz
cityhash-0.2.3.post9.tar.gz
cchardet-2.1.7.tar.gz
causalml-0.11.1.tar.gz
Cartopy-0.19.0.post1.tar.gz
av-8.0.3.tar.gz
asyncpg-0.24.0.tar.gz
astral-2.2.tar.gz
arch-5.0.1.tar.gz
arcgis-1.9.0.tar.gz
altgraph-0.17.tar.gz
aiokafka-0.7.1.tar.gz
aiohttp-3.7.4.post0.tar.gz
affinegap-1.11.tar.gz

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6RO2WFU5Q7UQDVL72IRMT4T6L4GEAKB6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

FWIW I've applied for an exception from the two-release deprecation policy
from the SC:
https://github.com/python/steering-council/issues/75

On Thu, Sep 2, 2021 at 1:12 AM Greg Ewing 
wrote:

> On 2/09/21 7:46 pm, Antoine Pitrou wrote:
> > Tracebacks are linked in a single direction, to go the other direction
> > you need to walk the frames attached to the traceback.
> >
> > So a (fake or not) frame object is still desirable, IMHO.
>
> Could we at least remove the necessity for a fake code object?
>
> --
> Greg
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TDLCJHNQSPNE7UXEJ33PV2VNQOPUFUT7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/25JN4NTFL5VAEB4F3RJO2RV6MQGQBXEB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Greg Ewing]

On 2/09/21 7:46 pm, Antoine Pitrou wrote:

Tracebacks are linked in a single direction, to go the other direction
you need to walk the frames attached to the traceback. 


So a (fake or not) frame object is still desirable, IMHO.


Could we at least remove the necessity for a fake code object?

--
Greg

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TDLCJHNQSPNE7UXEJ33PV2VNQOPUFUT7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Antoine Pitrou]

On Thu, 2 Sep 2021 13:31:32 +1200
Greg Ewing  wrote:
> On 2/09/21 4:46 am, Victor Stinner wrote:
> > If creating a fake frame is a common use case, we can maybe write a
> > public C API for that. For example, I saw parser injecting frames to
> > show the file name and line number of the parsed file in the
> > traceback.  
> 
> The way I would like to see this addressed is to make it possible
> to attach a filename and line number directly to a traceback object,
> without needing a frame or code object at all.

Tracebacks are linked in a single direction, to go the other direction
you need to walk the frames attached to the traceback.  If there is no
frame on the traceback, you cannot go the other direction.

So a (fake or not) frame object is still desirable, IMHO.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GJH33BQBHXMC6PGTHDY7TOQDRTCRXCCV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Petr Viktorin]

On 01. 09. 21 22:28, Guido van Rossum wrote:

I apologize, I keep making the same mistake.

The PyCode_New[WithPosArgs] functions are *not* in the stable ABI or in 
the limited API, so there's no need to petition the SC, nor do I need 
Petr's approval.


We may be bound by backwards compatibility for the *cpython* API, but I 
think that if Cython is okay if we just break this we should be fine. 
Users of the CPython API are expected to recompile for each new version, 
and if someone were to be using these functions with the old set of 
parameters the compiler would give them an error.


The cpython CPI is still covered by the backwards compatibility policy 
(PEP 387). You do need to ask the SC to skip the two-year deprecation 
period.


I don't see an issue with the exception being granted, but I do think it 
should be rubber-stamped as a project-wide decision.



So let's just choose (E) and d*mn backwards compatibility for these two 
functions.


That means:
- Get rid of PyCode_NewWithPosArgs altogether
- PyCode_New becomes unstable (and gets a new posinlyargcount argument)


... but still remains available and documented, just with a note that it 
may change in minor versions. Right?



On Wed, Sep 1, 2021 at 11:52 AM Guido van Rossum > wrote:


(context)

Guido van Rossum schrieb am 13.08.21 um 19:24:
 > In 3.11 we're changing a lot of details about code objects.
Part of this is
 > the "Faster CPython" work, part of it is other things (e.g.
PEP 657 -- Fine
 > Grained Error Locations in Tracebacks).
 >
 > As a result, the set of fields of the code object is
changing. This is
 > fine, the structure is part of the internal API anyway.
 >
 > But there's a problem with two public API functions,
PyCode_New() and
 > PyCode_NewWithPosArgs(). As we have them in the main (3.11)
branch, their
 > signatures are incompatible with previous versions, and they
have to be
 > since the set of values needed to create a code object is
different. (The
 > types.CodeType constructor signature is also changed, and so
is its
 > replace() method, but these aren't part of any stable API).
 >
 > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are
part of the PEP
 > 387 stable ABI. What should we do?
 >
 > A. We could deprecate them, keep (restore) their old
signatures, and create
 > crippled code objects (no exception table, no endline/column
tables,
 > qualname defaults to name).
 >
 > B. We could deprecate them, restore the old signatures, and
always raise an
 > error when they are called.
 >
 > C. We could just delete them.
 >
 > D. We could keep them, with modified signatures, and to heck
with ABI
 > compatibility for these two.
 >
 > E. We could get rid of PyCode_NewWithPosArgs(), update
PyCode() to add the
 > posonlyargcount (which is the only difference between the
two), and d*mn
 > the torpedoes.
 >
 > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
PyCode_New()
 > (and deprecate it).
 >
 > If these weren't part of the stable ABI, I'd choose (E). [...]


On Tue, Aug 31, 2021 at 7:07 PM Stefan Behnel mailto:stefan...@behnel.de>> wrote:

I also vote for (E). The creation of a code object is tied to
interpreter
internals and thus shouldn't be (or have been) declared stable.


I think you're one of the few people who call those functions, and
if even you think it's okay to break backward compatibility here, I
think we should just talk to the SC to be absolved of having these
two in the stable ABI. (Petr, do you agree? Without your backing I
don't feel comfortable even asking for this.)

I think the only problem with that argument is that code objects
are
required for frames. You could argue the same way about frames,
but then it
becomes really tricky to, you know, create frames for non-Python
code.


Note there's nothing in the stable ABI to create frames. There are
only functions to *get* an existing frame, to inspect a frame, and
to eval it. In any case even if there was a stable ABI function to
create a frame from a code object, one could argue that it's
sufficient to be able to get an existing code object from e.g. a
function object.

Since we're discussing this in the context of PEP 657, I wonder
if there's
a better way to create tracebacks from C code, other than
creating fake
frames with fake code objects.

Cython uses code objects and frames for the following use cases:

 

[Python-Dev] Re: Making code object APIs unstable

[Greg Ewing]

On 2/09/21 4:46 am, Victor Stinner wrote:

If creating a fake frame is a common use case, we can maybe write a
public C API for that. For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.


The way I would like to see this addressed is to make it possible
to attach a filename and line number directly to a traceback object,
without needing a frame or code object at all.

Creating a fake frame and code object just to do this is IMO an
ugly hack that should not be necessary.

--
Greg

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWX7EXJQ3JJRAL26YFR3GSSMLGSCEH2V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Nick Coghlan]

On Thu, 2 Sep 2021, 7:08 am Victor Stinner,  wrote:

> I saw Python projects injecting fake frames for XML and JSON parsers,
> maybe also configuration file (.ini?) parsers. So text files which
> have line numbers ;-)
>
> On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
>  wrote:
> > I don't think we should think on those terms. We certainly don't want to
> be on a case
> > where yet again we cannot change the internals because we have an
> official C-API exposed.
>
> PyCode_New() is annoying since it requires to provide *all* arguments.
> I'm thinking of an Frame API which only allows to set 2 values:
> filename and line number. Nothing else.
>
> Something like: frame = PyFrame_New(filename, lineno).
>


Perhaps "PyCode_FromLocation(file name, lineno)" for the new "good enough
to satisfy trace hooks and exception tracebacks" API? These are needed when
creating function-like (et al) objects from extension module code, not just
for runtime frames.

And then explicitly define which code object fields are expected to always
be populated and which are expected to be None on emulated code objects
that don't contain Python byte code? (e.g. some fields should be zero or
the empty tuple, rather than being set to None)

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AXGJAQ5MMRAKAKHHQMYVF5XS7PHALH4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

I saw Python projects injecting fake frames for XML and JSON parsers,
maybe also configuration file (.ini?) parsers. So text files which
have line numbers ;-)

On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
 wrote:
> I don't think we should think on those terms. We certainly don't want to be 
> on a case
> where yet again we cannot change the internals because we have an official 
> C-API exposed.

PyCode_New() is annoying since it requires to provide *all* arguments.
I'm thinking of an Frame API which only allows to set 2 values:
filename and line number. Nothing else.

Something like: frame = PyFrame_New(filename, lineno).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AII6O6ZXEENM7FJH3JRJIPTG4BRKJ3G5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

I apologize, I keep making the same mistake.

The PyCode_New[WithPosArgs] functions are *not* in the stable ABI or in the
limited API, so there's no need to petition the SC, nor do I need Petr's
approval.

We may be bound by backwards compatibility for the *cpython* API, but I
think that if Cython is okay if we just break this we should be fine. Users
of the CPython API are expected to recompile for each new version, and if
someone were to be using these functions with the old set of parameters the
compiler would give them an error.

So let's just choose (E) and d*mn backwards compatibility for these two
functions.

That means:
- Get rid of PyCode_NewWithPosArgs altogether
- PyCode_New becomes unstable (and gets a new posinlyargcount argument)

On Wed, Sep 1, 2021 at 11:52 AM Guido van Rossum  wrote:

> (context)
>
>> Guido van Rossum schrieb am 13.08.21 um 19:24:
>> > In 3.11 we're changing a lot of details about code objects. Part of
>> this is
>> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
>> Fine
>> > Grained Error Locations in Tracebacks).
>> >
>> > As a result, the set of fields of the code object is changing. This is
>> > fine, the structure is part of the internal API anyway.
>> >
>> > But there's a problem with two public API functions, PyCode_New() and
>> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch,
>> their
>> > signatures are incompatible with previous versions, and they have to be
>> > since the set of values needed to create a code object is different.
>> (The
>> > types.CodeType constructor signature is also changed, and so is its
>> > replace() method, but these aren't part of any stable API).
>> >
>> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
>> PEP
>> > 387 stable ABI. What should we do?
>> >
>> > A. We could deprecate them, keep (restore) their old signatures, and
>> create
>> > crippled code objects (no exception table, no endline/column tables,
>> > qualname defaults to name).
>> >
>> > B. We could deprecate them, restore the old signatures, and always
>> raise an
>> > error when they are called.
>> >
>> > C. We could just delete them.
>> >
>> > D. We could keep them, with modified signatures, and to heck with ABI
>> > compatibility for these two.
>> >
>> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
>> the
>> > posonlyargcount (which is the only difference between the two), and d*mn
>> > the torpedoes.
>> >
>> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
>> PyCode_New()
>> > (and deprecate it).
>> >
>> > If these weren't part of the stable ABI, I'd choose (E). [...]
>>
>
> On Tue, Aug 31, 2021 at 7:07 PM Stefan Behnel 
> wrote:
>
>> I also vote for (E). The creation of a code object is tied to interpreter
>> internals and thus shouldn't be (or have been) declared stable.
>>
>
> I think you're one of the few people who call those functions, and if even
> you think it's okay to break backward compatibility here, I think we should
> just talk to the SC to be absolved of having these two in the stable ABI.
> (Petr, do you agree? Without your backing I don't feel comfortable even
> asking for this.)
>
>
>> I think the only problem with that argument is that code objects are
>> required for frames. You could argue the same way about frames, but then
>> it
>> becomes really tricky to, you know, create frames for non-Python code.
>>
>
> Note there's nothing in the stable ABI to create frames. There are only
> functions to *get* an existing frame, to inspect a frame, and to eval it.
> In any case even if there was a stable ABI function to create a frame from
> a code object, one could argue that it's sufficient to be able to get an
> existing code object from e.g. a function object.
>
>
>> Since we're discussing this in the context of PEP 657, I wonder if
>> there's
>> a better way to create tracebacks from C code, other than creating fake
>> frames with fake code objects.
>>
>> Cython uses code objects and frames for the following use cases:
>>
>> - tracing generated C code at the Python syntax level
>> - profiling C-implemented functions
>> - tracebacks for C code
>>
>> Having a way to do these three efficiently (i.e. with close to zero
>> runtime
>> overhead) without having to reach into internals of the interpreter
>> state,
>> code objects and frames, would be nice.
>>
>> Failing that, I'm ok with declaring the relevant structs and C-API
>> functions non-stable and letting Cython use them as such, as we always
>> did.
>>
>
> I think others have answered this already -- in any case it's not the
> immediate subject of this thread, and I don't have a strong opinion on it.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

(context)

> Guido van Rossum schrieb am 13.08.21 um 19:24:
> > In 3.11 we're changing a lot of details about code objects. Part of this
> is
> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
> Fine
> > Grained Error Locations in Tracebacks).
> >
> > As a result, the set of fields of the code object is changing. This is
> > fine, the structure is part of the internal API anyway.
> >
> > But there's a problem with two public API functions, PyCode_New() and
> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
> > signatures are incompatible with previous versions, and they have to be
> > since the set of values needed to create a code object is different. (The
> > types.CodeType constructor signature is also changed, and so is its
> > replace() method, but these aren't part of any stable API).
> >
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
> PEP
> > 387 stable ABI. What should we do?
> >
> > A. We could deprecate them, keep (restore) their old signatures, and
> create
> > crippled code objects (no exception table, no endline/column tables,
> > qualname defaults to name).
> >
> > B. We could deprecate them, restore the old signatures, and always raise
> an
> > error when they are called.
> >
> > C. We could just delete them.
> >
> > D. We could keep them, with modified signatures, and to heck with ABI
> > compatibility for these two.
> >
> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
> the
> > posonlyargcount (which is the only difference between the two), and d*mn
> > the torpedoes.
> >
> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
> PyCode_New()
> > (and deprecate it).
> >
> > If these weren't part of the stable ABI, I'd choose (E). [...]
>

On Tue, Aug 31, 2021 at 7:07 PM Stefan Behnel  wrote:

> I also vote for (E). The creation of a code object is tied to interpreter
> internals and thus shouldn't be (or have been) declared stable.
>

I think you're one of the few people who call those functions, and if even
you think it's okay to break backward compatibility here, I think we should
just talk to the SC to be absolved of having these two in the stable ABI.
(Petr, do you agree? Without your backing I don't feel comfortable even
asking for this.)


> I think the only problem with that argument is that code objects are
> required for frames. You could argue the same way about frames, but then
> it
> becomes really tricky to, you know, create frames for non-Python code.
>

Note there's nothing in the stable ABI to create frames. There are only
functions to *get* an existing frame, to inspect a frame, and to eval it.
In any case even if there was a stable ABI function to create a frame from
a code object, one could argue that it's sufficient to be able to get an
existing code object from e.g. a function object.


> Since we're discussing this in the context of PEP 657, I wonder if there's
> a better way to create tracebacks from C code, other than creating fake
> frames with fake code objects.
>
> Cython uses code objects and frames for the following use cases:
>
> - tracing generated C code at the Python syntax level
> - profiling C-implemented functions
> - tracebacks for C code
>
> Having a way to do these three efficiently (i.e. with close to zero
> runtime
> overhead) without having to reach into internals of the interpreter state,
> code objects and frames, would be nice.
>
> Failing that, I'm ok with declaring the relevant structs and C-API
> functions non-stable and letting Cython use them as such, as we always did.
>

I think others have answered this already -- in any case it's not the
immediate subject of this thread, and I don't have a strong opinion on it.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z6VSLFMAHYBETSYEYGG4K7EA7DNW3L42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Pablo Galindo Salgado]

> If creating a fake frame is a common use case, we can maybe write a
public C API for that.

I don't think we should think on those terms. We certainly don't want to be
on a case
where yet again we cannot change the internals because we have an official
C-API exposed.


> For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.

You saw "parser" doing that? What is "parser"? Certainly is not the CPython
parser because I don't recall that we do any of that.


On Wed, 1 Sept 2021 at 17:48, Victor Stinner  wrote:

> If creating a fake frame is a common use case, we can maybe write a
> public C API for that. For example, I saw parser injecting frames to
> show the file name and line number of the parsed file in the
> traceback.
>
> Victor
>
> On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel  wrote:
> >
> > Guido van Rossum schrieb am 13.08.21 um 19:24:
> > > In 3.11 we're changing a lot of details about code objects. Part of
> this is
> > > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
> Fine
> > > Grained Error Locations in Tracebacks).
> > >
> > > As a result, the set of fields of the code object is changing. This is
> > > fine, the structure is part of the internal API anyway.
> > >
> > > But there's a problem with two public API functions, PyCode_New() and
> > > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch,
> their
> > > signatures are incompatible with previous versions, and they have to be
> > > since the set of values needed to create a code object is different.
> (The
> > > types.CodeType constructor signature is also changed, and so is its
> > > replace() method, but these aren't part of any stable API).
> > >
> > > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of
> the PEP
> > > 387 stable ABI. What should we do?
> > >
> > > A. We could deprecate them, keep (restore) their old signatures, and
> create
> > > crippled code objects (no exception table, no endline/column tables,
> > > qualname defaults to name).
> > >
> > > B. We could deprecate them, restore the old signatures, and always
> raise an
> > > error when they are called.
> > >
> > > C. We could just delete them.
> > >
> > > D. We could keep them, with modified signatures, and to heck with ABI
> > > compatibility for these two.
> > >
> > > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
> the
> > > posonlyargcount (which is the only difference between the two), and
> d*mn
> > > the torpedoes.
> > >
> > > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
> PyCode_New()
> > > (and deprecate it).
> > >
> > > If these weren't part of the stable ABI, I'd choose (E). [...]
> >
> > I also vote for (E). The creation of a code object is tied to interpreter
> > internals and thus shouldn't be (or have been) declared stable.
> >
> > I think the only problem with that argument is that code objects are
> > required for frames. You could argue the same way about frames, but then
> it
> > becomes really tricky to, you know, create frames for non-Python code.
> >
> > Since we're discussing this in the context of PEP 657, I wonder if
> there's
> > a better way to create tracebacks from C code, other than creating fake
> > frames with fake code objects.
> >
> > Cython uses code objects and frames for the following use cases:
> >
> > - tracing generated C code at the Python syntax level
> > - profiling C-implemented functions
> > - tracebacks for C code
> >
> > Having a way to do these three efficiently (i.e. with close to zero
> runtime
> > overhead) without having to reach into internals of the interpreter
> state,
> > code objects and frames, would be nice.
> >
> > Failing that, I'm ok with declaring the relevant structs and C-API
> > functions non-stable and letting Cython use them as such, as we always
> did.
> >
> > Stefan
> >
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QYDIQ5SS3FQCTDBBZ3JTI643KV4F6TNY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

If creating a fake frame is a common use case, we can maybe write a
public C API for that. For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.

Victor

On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel  wrote:
>
> Guido van Rossum schrieb am 13.08.21 um 19:24:
> > In 3.11 we're changing a lot of details about code objects. Part of this is
> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine
> > Grained Error Locations in Tracebacks).
> >
> > As a result, the set of fields of the code object is changing. This is
> > fine, the structure is part of the internal API anyway.
> >
> > But there's a problem with two public API functions, PyCode_New() and
> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
> > signatures are incompatible with previous versions, and they have to be
> > since the set of values needed to create a code object is different. (The
> > types.CodeType constructor signature is also changed, and so is its
> > replace() method, but these aren't part of any stable API).
> >
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP
> > 387 stable ABI. What should we do?
> >
> > A. We could deprecate them, keep (restore) their old signatures, and create
> > crippled code objects (no exception table, no endline/column tables,
> > qualname defaults to name).
> >
> > B. We could deprecate them, restore the old signatures, and always raise an
> > error when they are called.
> >
> > C. We could just delete them.
> >
> > D. We could keep them, with modified signatures, and to heck with ABI
> > compatibility for these two.
> >
> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the
> > posonlyargcount (which is the only difference between the two), and d*mn
> > the torpedoes.
> >
> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New()
> > (and deprecate it).
> >
> > If these weren't part of the stable ABI, I'd choose (E). [...]
>
> I also vote for (E). The creation of a code object is tied to interpreter
> internals and thus shouldn't be (or have been) declared stable.
>
> I think the only problem with that argument is that code objects are
> required for frames. You could argue the same way about frames, but then it
> becomes really tricky to, you know, create frames for non-Python code.
>
> Since we're discussing this in the context of PEP 657, I wonder if there's
> a better way to create tracebacks from C code, other than creating fake
> frames with fake code objects.
>
> Cython uses code objects and frames for the following use cases:
>
> - tracing generated C code at the Python syntax level
> - profiling C-implemented functions
> - tracebacks for C code
>
> Having a way to do these three efficiently (i.e. with close to zero runtime
> overhead) without having to reach into internals of the interpreter state,
> code objects and frames, would be nice.
>
> Failing that, I'm ok with declaring the relevant structs and C-API
> functions non-stable and letting Cython use them as such, as we always did.
>
> Stefan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QYDIQ5SS3FQCTDBBZ3JTI643KV4F6TNY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Stefan Behnel]

Guido van Rossum schrieb am 13.08.21 um 19:24:

In 3.11 we're changing a lot of details about code objects. Part of this is
the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine
Grained Error Locations in Tracebacks).

As a result, the set of fields of the code object is changing. This is
fine, the structure is part of the internal API anyway.

But there's a problem with two public API functions, PyCode_New() and
PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
signatures are incompatible with previous versions, and they have to be
since the set of values needed to create a code object is different. (The
types.CodeType constructor signature is also changed, and so is its
replace() method, but these aren't part of any stable API).

Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP
387 stable ABI. What should we do?

A. We could deprecate them, keep (restore) their old signatures, and create
crippled code objects (no exception table, no endline/column tables,
qualname defaults to name).

B. We could deprecate them, restore the old signatures, and always raise an
error when they are called.

C. We could just delete them.

D. We could keep them, with modified signatures, and to heck with ABI
compatibility for these two.

E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the
posonlyargcount (which is the only difference between the two), and d*mn
the torpedoes.

F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New()
(and deprecate it).

If these weren't part of the stable ABI, I'd choose (E). [...]


I also vote for (E). The creation of a code object is tied to interpreter 
internals and thus shouldn't be (or have been) declared stable.


I think the only problem with that argument is that code objects are 
required for frames. You could argue the same way about frames, but then it 
becomes really tricky to, you know, create frames for non-Python code.


Since we're discussing this in the context of PEP 657, I wonder if there's 
a better way to create tracebacks from C code, other than creating fake 
frames with fake code objects.


Cython uses code objects and frames for the following use cases:

- tracing generated C code at the Python syntax level
- profiling C-implemented functions
- tracebacks for C code

Having a way to do these three efficiently (i.e. with close to zero runtime 
overhead) without having to reach into internals of the interpreter state, 
code objects and frames, would be nice.


Failing that, I'm ok with declaring the relevant structs and C-API 
functions non-stable and letting Cython use them as such, as we always did.


Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Spencer Brown]

Victor Stinner wrote:
> Since Cython is a common consumer of this C API, can somone please dig
> into Cython to see exactly what it needs in terms of API? How does
> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
> copy an existing function to only override some fields, something like
> CodeType.replace(field=new_value)?
> If possible, I would prefer that Cython only uses the *public* C API.
> Otherwise, it will be very likely that Cython will break at every
> single Python release. Cython has a small team to maintain the code
> base, whereas CPython evolves much faster with a larger team.
> Victor

Cython only uses it here, to construct a fake code object with mostly blank 
values for each of the Cython function objects:
https://github.com/cython/cython/blob/8af0271186cc642436306274564986888d5e64c8/Cython/Compiler/ExprNodes.py#L9722
It's not actually intending to execute the code object at any point, it's 
created so that fake tracebacks get a filename/line number, the function 
objects have the argument count values for inspect to read and so Cython can 
emulate profiler/tracer events. So for Cython's case in particular it could a 
more stable API function, though said hypothetical API would have to have 
updated to handle pos-only args anyway also.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7RM4XJBH3KVSP3T43YCT6VCELTPCCBME/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

Both locations Greg found seem to be creating a completely *empty* code
object except for the name, filename and lineno. So that would actually
work fine with my proposal.

On Tue, Aug 17, 2021 at 4:24 PM Gregory P. Smith  wrote:

> +cc: cython-devel
>
> background reading for those new to the thread:
> https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/
>
> On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner 
> wrote:
>
>> Since Cython is a common consumer of this C API, can somone please dig
>> into Cython to see exactly what it needs in terms of API? How does
>> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
>> copy an existing function to only override some fields, something like
>> CodeType.replace(field=new_value)?
>>
>> If possible, I would prefer that Cython only uses the *public* C API.
>> Otherwise, it will be very likely that Cython will break at every
>> single Python release. Cython has a small team to maintain the code
>> base, whereas CPython evolves much faster with a larger team.
>>
>> Victor
>>
>
> I don't claim knowledge of Cython internals, but the two places it appears
> to call it's __Pyx_PyCode_New macro are:
>
>
> https://github.com/cython/cython/blob/master/Cython/Utility/Exceptions.c#L769
> in __Pyx_CreateCodeObjectForTraceback() - this one already has a `#if
> CYTHON_COMPILING_IN_LIMITED_API` code path option in it.
>
>  and
>
>
> https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L9722
> in CodeObjectNode.generate_result_code()  that creates PyCodeObject's for
> CyFunction instances per its comment.  Slightly described in this comment
> http://google3/third_party/py/cython/files/Cython/Compiler/ExprNodes.py?l=397.
> I don't see anything obvious mentioning the limited API in that code
> generator.
>
> it'd be best to loop in Cython maintainers for more of an idea of Cython's
> intents and needs with PyCode_New APIs.  I've cc'd cython-de...@python.org
> .
>
> -Greg
>
>
>> On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
>> >
>> > Doing a search of a huge codebase (work), the predominant user of
>> PyCode_New* APIs appears to be checked in Cython generated code (in all
>> sorts of third_party OSS projects). It's in the boilerplate that Cython
>> extensions make use of via it's __Pyx_PyCode_New macro.
>> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
>> >
>> > I saw very few non-Cython uses.  There are some, but at a very quick
>> first glance they appear simple - easy enough to reach out to the projects
>> with a PR to update their code.
>> >
>> > The Cython use will require people to upgrade Cython and regenerate
>> their code before they can use the Python version that changes these. That
>> is not an uncommon thing for Cython. It's unfortunate that many projects on
>> ship generated sources rather than use Cython at build time, but that isn't
>> _our_ problem to solve. The more often we change internal APIs that things
>> depend on, the more people will move their projects towards doing the right
>> thing with regards to either not using said APIs or rerunning an up to date
>> code generator as part of their build instead of checking in generated
>> unstable API using sources.
>> >
>> > -gps
>> >
>> >
>> > On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum 
>> wrote:
>> >>
>> >> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan 
>> wrote:
>> >>>
>> >>> [...]
>> >>> A cloning-with-replacement API that accepted the base code object and
>> the "safe to modify" fields could be a good complement to the API
>> deprecation proposal.
>> >>
>> >>
>> >> Yes (I forgot to mention that).
>> >>
>> >>>
>> >>> Moving actual "from scratch" code object creation behind the
>> Py_BUILD_CORE guard with an underscore prefix on the name would also make
>> sense, since it defines a key piece of the compiler/interpreter boundary.
>> >>
>> >>
>> >> Yeah, we have _PyCode_New() for that.
>> >>
>> >>>
>> >>> Cheers,
>> >>> Nick.
>> >>>
>> >>> P.S. Noting an idea that won't work, in case anyone else reading the
>> thread was thinking the same thing: a "PyType_FromSpec" style API won't
>> help here, as the issue is that the compiler is now doing more work up
>> front and recording that extra info in the code object for the interpreter
>> to use. There is no way to synthesise that info if it isn't passed to the
>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>> >>
>> >>
>> >> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
>> pushed for this and implemented it). You gave me an idea now: the C
>> equivalent to .replace() could use the same input structure; one can leave
>> fields NULL that should be copied from the original unmodified.
>> >>
>> >> --
>> >> --Guido van Rossum (python.org/~guido)
>> >> Pronouns: he/him (why is my pronoun here?)
>> >> ___
>> >> Python-Dev mailing list -- 

[Python-Dev] Re: Making code object APIs unstable

[Gregory P. Smith]

+cc: cython-devel

background reading for those new to the thread:
https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/

On Tue, Aug 17, 2021 at 9:47 AM Victor Stinner  wrote:

> Since Cython is a common consumer of this C API, can somone please dig
> into Cython to see exactly what it needs in terms of API? How does
> Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
> copy an existing function to only override some fields, something like
> CodeType.replace(field=new_value)?
>
> If possible, I would prefer that Cython only uses the *public* C API.
> Otherwise, it will be very likely that Cython will break at every
> single Python release. Cython has a small team to maintain the code
> base, whereas CPython evolves much faster with a larger team.
>
> Victor
>

I don't claim knowledge of Cython internals, but the two places it appears
to call it's __Pyx_PyCode_New macro are:

https://github.com/cython/cython/blob/master/Cython/Utility/Exceptions.c#L769
in __Pyx_CreateCodeObjectForTraceback() - this one already has a `#if
CYTHON_COMPILING_IN_LIMITED_API` code path option in it.

 and

https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L9722
in CodeObjectNode.generate_result_code()  that creates PyCodeObject's for
CyFunction instances per its comment.  Slightly described in this comment
http://google3/third_party/py/cython/files/Cython/Compiler/ExprNodes.py?l=397.
I don't see anything obvious mentioning the limited API in that code
generator.

it'd be best to loop in Cython maintainers for more of an idea of Cython's
intents and needs with PyCode_New APIs.  I've cc'd cython-de...@python.org.

-Greg


> On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
> >
> > Doing a search of a huge codebase (work), the predominant user of
> PyCode_New* APIs appears to be checked in Cython generated code (in all
> sorts of third_party OSS projects). It's in the boilerplate that Cython
> extensions make use of via it's __Pyx_PyCode_New macro.
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
> >
> > I saw very few non-Cython uses.  There are some, but at a very quick
> first glance they appear simple - easy enough to reach out to the projects
> with a PR to update their code.
> >
> > The Cython use will require people to upgrade Cython and regenerate
> their code before they can use the Python version that changes these. That
> is not an uncommon thing for Cython. It's unfortunate that many projects on
> ship generated sources rather than use Cython at build time, but that isn't
> _our_ problem to solve. The more often we change internal APIs that things
> depend on, the more people will move their projects towards doing the right
> thing with regards to either not using said APIs or rerunning an up to date
> code generator as part of their build instead of checking in generated
> unstable API using sources.
> >
> > -gps
> >
> >
> > On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum 
> wrote:
> >>
> >> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan 
> wrote:
> >>>
> >>> [...]
> >>> A cloning-with-replacement API that accepted the base code object and
> the "safe to modify" fields could be a good complement to the API
> deprecation proposal.
> >>
> >>
> >> Yes (I forgot to mention that).
> >>
> >>>
> >>> Moving actual "from scratch" code object creation behind the
> Py_BUILD_CORE guard with an underscore prefix on the name would also make
> sense, since it defines a key piece of the compiler/interpreter boundary.
> >>
> >>
> >> Yeah, we have _PyCode_New() for that.
> >>
> >>>
> >>> Cheers,
> >>> Nick.
> >>>
> >>> P.S. Noting an idea that won't work, in case anyone else reading the
> thread was thinking the same thing: a "PyType_FromSpec" style API won't
> help here, as the issue is that the compiler is now doing more work up
> front and recording that extra info in the code object for the interpreter
> to use. There is no way to synthesise that info if it isn't passed to the
> constructor, as it isn't intrinsically recorded in the opcode sequence.
> >>
> >>
> >> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
> pushed for this and implemented it). You gave me an idea now: the C
> equivalent to .replace() could use the same input structure; one can leave
> fields NULL that should be copied from the original unmodified.
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido)
> >> Pronouns: he/him (why is my pronoun here?)
> >> ___
> >> Python-Dev mailing list -- python-dev@python.org
> >> To unsubscribe send an email to python-dev-le...@python.org
> >> https://mail.python.org/mailman3/lists/python-dev.python.org/
> >> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> > 

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

Since Cython is a common consumer of this C API, can somone please dig
into Cython to see exactly what it needs in terms of API? How does
Cython create all arguments of the __Pyx_PyCode_New() macro? Does it
copy an existing function to only override some fields, something like
CodeType.replace(field=new_value)?

If possible, I would prefer that Cython only uses the *public* C API.
Otherwise, it will be very likely that Cython will break at every
single Python release. Cython has a small team to maintain the code
base, whereas CPython evolves much faster with a larger team.

Victor

On Tue, Aug 17, 2021 at 8:51 AM Gregory P. Smith  wrote:
>
> Doing a search of a huge codebase (work), the predominant user of PyCode_New* 
> APIs appears to be checked in Cython generated code (in all sorts of 
> third_party OSS projects). It's in the boilerplate that Cython extensions 
> make use of via it's __Pyx_PyCode_New macro.  
> https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470
>
> I saw very few non-Cython uses.  There are some, but at a very quick first 
> glance they appear simple - easy enough to reach out to the projects with a 
> PR to update their code.
>
> The Cython use will require people to upgrade Cython and regenerate their 
> code before they can use the Python version that changes these. That is not 
> an uncommon thing for Cython. It's unfortunate that many projects on ship 
> generated sources rather than use Cython at build time, but that isn't _our_ 
> problem to solve. The more often we change internal APIs that things depend 
> on, the more people will move their projects towards doing the right thing 
> with regards to either not using said APIs or rerunning an up to date code 
> generator as part of their build instead of checking in generated unstable 
> API using sources.
>
> -gps
>
>
> On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum  wrote:
>>
>> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:
>>>
>>> [...]
>>> A cloning-with-replacement API that accepted the base code object and the 
>>> "safe to modify" fields could be a good complement to the API deprecation 
>>> proposal.
>>
>>
>> Yes (I forgot to mention that).
>>
>>>
>>> Moving actual "from scratch" code object creation behind the Py_BUILD_CORE 
>>> guard with an underscore prefix on the name would also make sense, since it 
>>> defines a key piece of the compiler/interpreter boundary.
>>
>>
>> Yeah, we have _PyCode_New() for that.
>>
>>>
>>> Cheers,
>>> Nick.
>>>
>>> P.S. Noting an idea that won't work, in case anyone else reading the thread 
>>> was thinking the same thing: a "PyType_FromSpec" style API won't help here, 
>>> as the issue is that the compiler is now doing more work up front and 
>>> recording that extra info in the code object for the interpreter to use. 
>>> There is no way to synthesise that info if it isn't passed to the 
>>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>>
>>
>> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC pushed 
>> for this and implemented it). You gave me an idea now: the C equivalent to 
>> .replace() could use the same input structure; one can leave fields NULL 
>> that should be copied from the original unmodified.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> Pronouns: he/him (why is my pronoun here?)
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/67DMIW7NQE6M6LEPLANXKZQEFOFVPBBL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FFE6ULML5CWOBMFCZKTRCSSHKTPBFRFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Petr Viktorin]

I'm late to the thread, and as I read it I see everything I wanted to 
say was covered already :)

So just a few clarifications.

The stable ABI is not defined by PEP 384 or PEP 652 or by the header 
something is defined in, but by the docs:

- https://docs.python.org/dev/c-api/stable.html
and changes to it are covered in the devguide:
- https://devguide.python.org/c-api/


We have 3 API layers:

1. Internal API, guarded by Py_BUILD_CORE, can break *even in point 
releases*. (Py_BUILD_CORE means just that: things like `_PyCode_New` can 
only be used safely if you build/embed CPython yourself.)
2. Regular C-API, covered by PEP 387 (breaking changes need deprecation 
for 2 releases, or an exception from the SC); `PyCode_New*` is here now

2. Stable ABI, which is hard to change, and thankfully isn't involved here.

I can see that having `.replace()` equivalent in the C API would be 
"worth the effort of [its users having to] keeping up with CPython 
internal changes" (to quote Patrick).
Looks like we could use something between layers 1 and 2 above for 
"high-maintenance" users (like Cython): API that will work for all of 
3.11.x, but can freely break for 3.12. I don't think this needs an 
explicit API layer, though: just a note in the docs that a new 
`PyCode_NewWithAllTheWithBellsAndWhistles` is expected to change in 
point releases. But...


Guido:

  [struct rather than N arguments] is the
  API style that _PyCode_New() uses (thanks to Eric who
  IIRC pushed for this and implemented it). You gave me an idea now:
  the C equivalent to .replace() could use the same input structure;
  one can leave fields NULL that should be copied from the original
  unmodified.


From a usability point of view, that's a much better idea than a 
function that's expected to change. It would probably also be easier to 
implement than an entirely separate public API.


Nick:

  P.S. Noting an idea that won't work, in case anyone else reading
  the thread was thinking the same thing: a "PyType_FromSpec"
  style API won't help here, as the issue is that the compiler is
  now doing more work up front and recording that extra info in
  the code object for the interpreter to use. There is no way to
  synthesise that info if it isn't passed to the constructor, as
  it isn't intrinsically recorded in the opcode sequence.


I guess it might be possible to add a flag that says a piece of bytecode 
object has exception handling and so it needs an exception table, and 
have the old API raise when the flag is on.

It's probably not worth the effort, though.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T32UC25S3R7MTAKTZRF22ZJ26K6WMFGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Gregory P. Smith]

Doing a search of a huge codebase (work), the predominant user of
PyCode_New* APIs appears to be checked in Cython generated code (in all
sorts of third_party OSS projects). It's in the boilerplate that Cython
extensions make use of via it's __Pyx_PyCode_New macro.
https://github.com/cython/cython/blob/master/Cython/Utility/ModuleSetupCode.c#L470

I saw very few non-Cython uses.  There are some, but at a very quick first
glance they appear simple - easy enough to reach out to the projects with a
PR to update their code.

The Cython use will require people to upgrade Cython and regenerate their
code before they can use the Python version that changes these. That is not
an uncommon thing for Cython. It's unfortunate that many projects on ship
generated sources rather than use Cython at build time, but that isn't
_our_ problem to solve. The more often we change internal APIs that things
depend on, the more people will move their projects towards doing the right
thing with regards to either not using said APIs or rerunning an up to date
code generator as part of their build instead of checking in generated
unstable API using sources.

-gps


On Mon, Aug 16, 2021 at 8:04 PM Guido van Rossum  wrote:

> On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:
>
>> [...]
>> A cloning-with-replacement API that accepted the base code object and the
>> "safe to modify" fields could be a good complement to the API deprecation
>> proposal.
>>
>
> Yes (I forgot to mention that).
>
>
>> Moving actual "from scratch" code object creation behind the
>> Py_BUILD_CORE guard with an underscore prefix on the name would also make
>> sense, since it defines a key piece of the compiler/interpreter boundary.
>>
>
> Yeah, we have _PyCode_New() for that.
>
>
>> Cheers,
>> Nick.
>>
>> P.S. Noting an idea that won't work, in case anyone else reading the
>> thread was thinking the same thing: a "PyType_FromSpec" style API won't
>> help here, as the issue is that the compiler is now doing more work up
>> front and recording that extra info in the code object for the interpreter
>> to use. There is no way to synthesise that info if it isn't passed to the
>> constructor, as it isn't intrinsically recorded in the opcode sequence.
>>
>
> That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
> pushed for this and implemented it). You gave me an idea now: the C
> equivalent to .replace() could use the same input structure; one can leave
> fields NULL that should be copied from the original unmodified.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/67DMIW7NQE6M6LEPLANXKZQEFOFVPBBL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Mon, Aug 16, 2021 at 4:44 PM Nick Coghlan  wrote:

> [...]
> A cloning-with-replacement API that accepted the base code object and the
> "safe to modify" fields could be a good complement to the API deprecation
> proposal.
>

Yes (I forgot to mention that).


> Moving actual "from scratch" code object creation behind the Py_BUILD_CORE
> guard with an underscore prefix on the name would also make sense, since it
> defines a key piece of the compiler/interpreter boundary.
>

Yeah, we have _PyCode_New() for that.


> Cheers,
> Nick.
>
> P.S. Noting an idea that won't work, in case anyone else reading the
> thread was thinking the same thing: a "PyType_FromSpec" style API won't
> help here, as the issue is that the compiler is now doing more work up
> front and recording that extra info in the code object for the interpreter
> to use. There is no way to synthesise that info if it isn't passed to the
> constructor, as it isn't intrinsically recorded in the opcode sequence.
>

That's the API style that _PyCode_New() uses (thanks to Eric who IIRC
pushed for this and implemented it). You gave me an idea now: the C
equivalent to .replace() could use the same input structure; one can leave
fields NULL that should be copied from the original unmodified.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NWYMCDAMS4YRJ7ESXNWQ6MIBSRAZEXEM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Nick Coghlan]

On Tue, 17 Aug 2021, 4:30 am Guido van Rossum,  wrote:

> On Mon, Aug 16, 2021 at 9:30 AM Steve Dower 
> wrote:
>
>> On 8/16/2021 12:47 AM, Guido van Rossum wrote:
>> > My current proposal is to issue a DeprecationWarning in PyCode_New()
>> and
>> > PyCode_NewWithPosArgs(), which can be turned into an error using a
>> > command-line flag. If it's made an error, we effectively have (B); by
>> > default, we have (A).
>> >
>> > Then in 3.13 we can drop them completely.
>>
>> We definitely had legitimate use cases come up when adding positional
>> arguments (hence the new API, rather than breaking the existing one,
>> which was the first attempt at adding the feature).
>>
>> I don't recall exactly what they are (perhaps Pablo does, or they may be
>> in email/issue archives), but since they exist, presumably they are
>> useful and viable _despite_ the bytecode varying between releases. This
>> suggests there's probably a better API we should add at the same time -
>> possibly some kind of unmarshalling or cloning-with-updates function?
>>
>
> I presume the use cases are essentially some variant of the .replace() API
> that exists at the Python level. At the C level you would get all fields
> from an existing code object and pass them to PyCode_New[WithPosArgs]
> except for e.g. the co_filename field. Unfortunately those use cases will
> still break if there are any try blocks in the code or if the
> endline/column info is needed. Also you can't access any of the code
> object's fields without the internal API (this wasn't always so). So I
> think it's different now.
>

A cloning-with-replacement API that accepted the base code object and the
"safe to modify" fields could be a good complement to the API deprecation
proposal.

Moving actual "from scratch" code object creation behind the Py_BUILD_CORE
guard with an underscore prefix on the name would also make sense, since it
defines a key piece of the compiler/interpreter boundary.

Cheers,
Nick.

P.S. Noting an idea that won't work, in case anyone else reading the thread
was thinking the same thing: a "PyType_FromSpec" style API won't help here,
as the issue is that the compiler is now doing more work up front and
recording that extra info in the code object for the interpreter to use.
There is no way to synthesise that info if it isn't passed to the
constructor, as it isn't intrinsically recorded in the opcode sequence.


>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/K2PUYK6MHXILWCBDWZZIUORIALI6SP4A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YIKXHLNR4FYP7ZU5PUTRIBNHENIXEM7A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Mon, Aug 16, 2021 at 9:30 AM Steve Dower  wrote:

> On 8/16/2021 12:47 AM, Guido van Rossum wrote:
> > My current proposal is to issue a DeprecationWarning in PyCode_New() and
> > PyCode_NewWithPosArgs(), which can be turned into an error using a
> > command-line flag. If it's made an error, we effectively have (B); by
> > default, we have (A).
> >
> > Then in 3.13 we can drop them completely.
>
> We definitely had legitimate use cases come up when adding positional
> arguments (hence the new API, rather than breaking the existing one,
> which was the first attempt at adding the feature).
>
> I don't recall exactly what they are (perhaps Pablo does, or they may be
> in email/issue archives), but since they exist, presumably they are
> useful and viable _despite_ the bytecode varying between releases. This
> suggests there's probably a better API we should add at the same time -
> possibly some kind of unmarshalling or cloning-with-updates function?
>

I presume the use cases are essentially some variant of the .replace() API
that exists at the Python level. At the C level you would get all fields
from an existing code object and pass them to PyCode_New[WithPosArgs]
except for e.g. the co_filename field. Unfortunately those use cases will
still break if there are any try blocks in the code or if the
endline/column info is needed. Also you can't access any of the code
object's fields without the internal API (this wasn't always so). So I
think it's different now.


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K2PUYK6MHXILWCBDWZZIUORIALI6SP4A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Steve Dower]

On 8/16/2021 12:47 AM, Guido van Rossum wrote:
My current proposal is to issue a DeprecationWarning in PyCode_New() and 
PyCode_NewWithPosArgs(), which can be turned into an error using a 
command-line flag. If it's made an error, we effectively have (B); by 
default, we have (A).


Then in 3.13 we can drop them completely.


We definitely had legitimate use cases come up when adding positional 
arguments (hence the new API, rather than breaking the existing one, 
which was the first attempt at adding the feature).


I don't recall exactly what they are (perhaps Pablo does, or they may be 
in email/issue archives), but since they exist, presumably they are 
useful and viable _despite_ the bytecode varying between releases. This 
suggests there's probably a better API we should add at the same time - 
possibly some kind of unmarshalling or cloning-with-updates function?


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NXSTTN3CVCGAYSCUH24OEZZFIBOEJRBD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Sat, Aug 14, 2021 at 4:56 AM Serhiy Storchaka 
wrote:

> 13.08.21 20:24, Guido van Rossum пише:
> > If these weren't part of the stable ABI, I'd choose (E). But because
> > they are, I think only (A) or (B) are our options. The problem with (C)
> > is that if there's code that links to them but doesn't call them (except
> > in some corner case that the user can avoid), the code won't link even
> > though it would work fine. The problem with (D) is that if it *is*
> > called by code expecting the old signature it will segfault. I'm not
> > keen on (A) since it can cause broken code objects when used to copy a
> > code object with some modified metadata (e.g. a different filename),
> > since there's no way to pass the exception table (and several other
> > fields, but the exception table is an integral part of the code now).
>
> I agree that (A) and (B) are only options if we preserve binary
> compatibility. For practical reasons I prefer (B).
>
> We can make (A) working if add the exception table to the end of the
> bytecode array and the endline/column tables to the end of the lineno
> table. It would allow to re-construct the code object with some simple
> changes (like filename or replace some constants). Creating the code
> object from zero is version-specific in any case, because bytecode is
> changed in every version, and semantic of some fields can be changed too
> (e.g. support of negative offsets in the lineno table). But it would
> complicate the code object structure and the code that works with it in
> long term.
>

That sounds like a perversion of backward compatibility. The endline and
column tables are already optional (there's a flag to suppress them and
then the fields are set to None) and we can just not support code that
catches exceptions. If you take e.g.

def f(x):
try:
1/0
except:
print("NO")

and you remove the exception table you get code that is equivalent to this:

def f(x):
1/0

plus some unreachable bytecode.

My current proposal is to issue a DeprecationWarning in PyCode_New() and
PyCode_NewWithPosArgs(), which can be turned into an error using a
command-line flag. If it's made an error, we effectively have (B); by
default, we have (A).

Then in 3.13 we can drop them completely.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HVVBTYUM3EYBVYSDBGABDVTC5JBG74VS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Jim J. Jewett]

Guido van Rossum wrote:
> On Fri, Aug 13, 2021 at 11:17 AM Terry Reedy tjre...@udel.edu wrote:
> > On 8/13/2021 1:24 PM, Guido van Rossum wrote:

> I'm actually not sure what use case you're talking about.

"source/code that hadn't actually changed and was just being re-compiled and 
run".  I've done it as the first stage of porting or reviving long-dead code.  
I've seen it done when deployment is handed off to a different group than 
development, and also when using a library component that had not been recently 
maintained.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B2VROA577DHONXYE3OL5XXKKLLRUNX6R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Serhiy Storchaka]

13.08.21 20:24, Guido van Rossum пише:
> If these weren't part of the stable ABI, I'd choose (E). But because
> they are, I think only (A) or (B) are our options. The problem with (C)
> is that if there's code that links to them but doesn't call them (except
> in some corner case that the user can avoid), the code won't link even
> though it would work fine. The problem with (D) is that if it *is*
> called by code expecting the old signature it will segfault. I'm not
> keen on (A) since it can cause broken code objects when used to copy a
> code object with some modified metadata (e.g. a different filename),
> since there's no way to pass the exception table (and several other
> fields, but the exception table is an integral part of the code now).

I agree that (A) and (B) are only options if we preserve binary
compatibility. For practical reasons I prefer (B).

We can make (A) working if add the exception table to the end of the
bytecode array and the endline/column tables to the end of the lineno
table. It would allow to re-construct the code object with some simple
changes (like filename or replace some constants). Creating the code
object from zero is version-specific in any case, because bytecode is
changed in every version, and semantic of some fields can be changed too
(e.g. support of negative offsets in the lineno table). But it would
complicate the code object structure and the code that works with it in
long term.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/B4OGG4EQUB4LPRLQRBZ2RBX4DVQJILST/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Terry Reedy]

On 8/13/2021 8:45 PM, Guido van Rossum wrote:

Now, backwards compatibility is still nothing to sneeze at, but at least 
we don't have to hem and haw about ABI compatibility.


If back compatibility were our sacred prime directive, then we would not 
(or should not) have changed code objects in way that threatens it.


The question then is, can we break (source) backwards compatibility for 
these two functions, because in practice code objects have been 
unstable?


I presume that the existence of those functions is based on an 
expectation of stability that may have been true for years, but not 
recently.  Or an expectation that any change would be part of a one-time 
overhaul where compatibility was suspended as much as for 3.0.  I think 
that whatever we do now should allow for continued version-by-version 
changes in the future.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKHISUP624VI7MVUG7XPYO5QTBLFIUT2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Fri, Aug 13, 2021 at 11:17 AM Terry Reedy  wrote:

> On 8/13/2021 1:24 PM, Guido van Rossum wrote:
> [...]
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
> > PEP 387 stable ABI. What should we do?
>
> PEP 387 is Backwards Compatibility Policy
> https://www.python.org/dev/peps/pep-0387/
> Did you mean PEP 384 -- Defining a Stable ABI
> https://www.python.org/dev/peps/pep-0384/
>

Aargh! You're right! I misremembered the PEP number (and didn't double
check) and when Pablo said we were "constrained by PEP 387" I assumed he
meant the stable ABI. Sorry for the goof-up.

Now, backwards compatibility is still nothing to sneeze at, but at least we
don't have to hem and haw about ABI compatibility.

The question then is, can we break (source) backwards compatibility for
these two functions, because in practice code objects have been unstable?
Or do we need to go the deprecation route here using (A) or (B) for two
releases. Presumably we could combine these two so that it reports a
warning when called, and if the warning has been configured to be replaced
with an error it will raise that.

Thanks also to Eric for pointing out the same thing. (I should probably
have waited to discuss this with him before posting. Sorry again.)


On Fri, Aug 13, 2021 at 3:39 PM Jim J. Jewett  wrote:

> How badly would the code objects be crippled?
>
> >(no exception table, no endline/column tables,
> > qualname defaults to name)
>
> That sounds like it would be a pain for debugging, but might still work
> for source/code that hadn't actually changed and was just being re-compiled
> and run (possibly with the caveat that the data needs to be clean enough to
> avoid exceptions).
>

Yeah, I would be okay with faking it for endline/column tables and
qualname, but I balk at the exception table -- this is used for the new
3.11 concept "zero cost exception handling" where a try block doesn't
require any setup (the opcode for this has gone away). Too much valid code
catches exceptions (e.g. KeyError) to trust that this will mostly do the
right thing.


> Since that is a common use case, and one where there is a good reason not
> to make any source-level changes, it would be good to keep compatibility
> for that minimal level.


I'm actually not sure what use case you're talking about.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OVXUGWFGDLMKKW7VZFFUYEU7ZNHAROVQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Jim J. Jewett]

How badly would the code objects be crippled?

>(no exception table, no endline/column tables,
> qualname defaults to name)

That sounds like it would be a pain for debugging, but might still work for 
source/code that hadn't actually changed and was just being re-compiled and run 
(possibly with the caveat that the data needs to be clean enough to avoid 
exceptions).

Since that is a common use case, and one where there is a good reason not to 
make any source-level changes, it would be good to keep compatibility for that 
minimal level.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/URGJJWVEDKROLYOUWQ5QQ5CROZHLTH7G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Eric Snow]

On Fri, Aug 13, 2021 at 11:29 AM Guido van Rossum  wrote:
> If these weren't part of the stable ABI, I'd choose (E).

They aren't in the stable ABI (or limited API).  Instead, they are
part of the broader public API (declared in Include/cpython/code.h,
along with "struct PyCodeObject" and others).

FWIW, there is actually very little API related to PyCodeObject that
is in the limited API:

* Include/code.h:typedef struct PyCodeObject PyCodeObject;
* Include/genobject.h:PyCodeObject *prefix##_code;
* Include/pyframe.h:PyAPI_FUNC(PyCodeObject *)
PyFrame_GetCode(PyFrameObject *frame);

All that said, the issue of compatibility remains.  I mostly agree
with Guido's analysis and his choice of (E), as long as it's
appropriately documented as unstable.

However, I'd probably pick (C) with a caveat.  We already have a
classification for this sort of unstable API: "internal".  Given how
code objects are so coupled to the CPython internals, I suggest that
most API related to PyCodeObject belongs in the internal API (in
Include/internal/pycore_code.h) and thus moved out of the public API.
Folks that are creating code objects manually via the C-API are
probably already doing low-level stuff that requires other "internal"
API (via Py_BUILD_CORE, etc.).  Otherwise they should use
types.CodeType instead.

Making that change would naturally include dropping PyCode_New() and
PyCode_NewWithPosArgs(), as described in (C).  However, we already
have _PyCode_New() in the internal API.  (It is slightly different but
effectively equivalent.)  We could either drop the underscore on
_PyCode_New() or move the existing PyCode_NewWithPosArgs() (renamed to
PyCode_New) to live beside it.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KWWRLL56EI2S5BVADKMDCG4UED76GXXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Patrick Reader]

I'm a major consumer of these APIs as part of some commercial projects 
(unfortunately I can't discuss too much further) but we find it worth the 
effort of keeping up with CPython internal changes to continue using them. 
Option D seems like the best option from my point of view; any user would need 
to be able to keep up with bytecode changes already so I think backwards 
compatibility isn't really a concern.

Unless you feel it's extremely important to follow PEP 387 (slippery slope?), 
I'd just say "to hell with it". Also, at least if it segfaults, people who 
haven't updated their code will know very quickly, whereas some other options 
might cause subtler bugs.

On 13/08/2021 18:24, Guido van Rossum wrote:
> In 3.11 we're changing a lot of details about code objects. Part of this is 
> the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine 
> Grained Error Locations in Tracebacks).
>
> As a result, the set of fields of the code object is changing. This is fine, 
> the structure is part of the internal API anyway.
>
> But there's a problem with two public API functions, PyCode_New() and 
> PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their 
> signatures are incompatible with previous versions, and they have to be since 
> the set of values needed to create a code object is different. (The 
> types.CodeType constructor signature is also changed, and so is its replace() 
> method, but these aren't part of any stable API).
>
> Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP 
> 387 stable ABI. What should we do?
>
> A. We could deprecate them, keep (restore) their old signatures, and create 
> crippled code objects (no exception table, no endline/column tables, qualname 
> defaults to name).
>
> B. We could deprecate them, restore the old signatures, and always raise an 
> error when they are called.
>
> C. We could just delete them.
>
> D. We could keep them, with modified signatures, and to heck with ABI 
> compatibility for these two.
>
> E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the 
> posonlyargcount (which is the only difference between the two), and d*mn the 
> torpedoes.
>
> F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New() 
> (and deprecate it).
>
> If these weren't part of the stable ABI, I'd choose (E). But because they 
> are, I think only (A) or (B) are our options. The problem with (C) is that if 
> there's code that links to them but doesn't call them (except in some corner 
> case that the user can avoid), the code won't link even though it would work 
> fine. The problem with (D) is that if it *is* called by code expecting the 
> old signature it will segfault. I'm not keen on (A) since it can cause broken 
> code objects when used to copy a code object with some modified metadata 
> (e.g. a different filename), since there's no way to pass the exception table 
> (and several other fields, but the exception table is an integral part of the 
> code now).
>
> Code wanting to make slight modifications to code objects such as changing 
> co_name or co_filename should switch to the .replace() API, which is much 
> better at this (though calling it from C is a pain, it's possible). Code 
> wanting to synthesize code should be updated for each release; we should 
> probably require such code to be built with the internal API and use 
> _PyCode_New(), which takes a struct argument containing all the necessary 
> fields.
>
> Thoughts? I'm especially interested in Petr's opinion given that this is a 
> case where we'd like to deprecate something in the stable ABI.
>
> See also discussion in https://bugs.python.org/issue40222 
>  (esp. near the end).
>
> -- 
> --Guido van Rossum (python.org/~guido )
> /Pronouns: he/him //(why is my pronoun here?)/ 
> 
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NLGBBH6ZPW6O5K2MAUIXEGKLYCAU6NHK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Terry Reedy]

On 8/13/2021 1:24 PM, Guido van Rossum wrote:
In 3.11 we're changing a lot of details about code objects. Part of this 
is the "Faster CPython" work, part of it is other things (e.g. PEP 657 
-- Fine Grained Error Locations in Tracebacks).


As a result, the set of fields of the code object is changing. This is 
fine, the structure is part of the internal API anyway.


But there's a problem with two public API functions, PyCode_New() and 
PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, 
their signatures are incompatible with previous versions, and they have 
to be since the set of values needed to create a code object is 
different. (The types.CodeType constructor signature is also changed, 
and so is its replace() method, but these aren't part of any stable API).


Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the 
PEP 387 stable ABI. What should we do?


PEP 387 is Backwards Compatibility Policy
https://www.python.org/dev/peps/pep-0387/
Did you mean PEP 384 -- Defining a Stable ABI
https://www.python.org/dev/peps/pep-0384/

How are PyCode_xxx included?  384 defines code objects as 'internal 
objects'.  "In some cases, the incompatibilities only affect internal 
objects of the interpreter, such as frame or code objects."  And Firefox 
does not find 'pycode'.


A. We could deprecate them, keep (restore) their old signatures, and 
create crippled code objects (no exception table, no endline/column 
tables, qualname defaults to name).


B. We could deprecate them, restore the old signatures, and always raise 
an error when they are called.


These seem pretty useless.


C. We could just delete them.

D. We could keep them, with modified signatures, and to heck with ABI 
compatibility for these two.


E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add 
the posonlyargcount (which is the only difference between the two), and 
d*mn the torpedoes.


F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for 
PyCode_New() (and deprecate it).



--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3OXTGNT5DTBJBWF7ZXLQB22NYNZWHJI7/
Code of Conduct: http://python.org/psf/codeofconduct/