[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Brett Cannon]

On Mon, May 30, 2022 at 12:54 PM Steve Dower  wrote:

> I prefer separate header files, provided people outside of core always
> have one (presumably "Python.h") that should be included first and
> includes enough info to check which headers will be available (i.e. the
> version defs).
>

The idea we were kicking around was e.g. `Python-unstable.h` would be all
of the limited API plus the unstable parts, `Python-unlimited.h` would be
**everything**, etc. I would expect `Python.h` would continue to be what it
is today for compatibility purposes. There wouldn't necessarily be an
"always have one" header since these header files would cascade into each
other as you opted into more and more unstable APIs (think about this as
layers of APIs  and representing each encompassing layer with a header
file). This would also let teams set policies of how much instability risk
they were willing to take by having CI have an allowlist/blocklist of
Python header files.

-Brett


>
> Modifying preprocessor definitions for different Python versions, or
> having to set them before knowing what version is being used, seems more
> complicated.
>

> Cheers,
> Steve
>
> On 5/30/2022 8:26 PM, Brett Cannon wrote:
> > We discussed having leading underscores for this API tier, and it was
> decided that a leading underscore was preferred.
> >
> > This did start a discussion, though, about whether we should control API
> access/opt-in via `#include` by having `.h` files that convey what API the
> user is opting into, or use `#define` to control what gets exposed via
> `Python.h`. The general feeling was that the header file idea is ideal, but
> it is a little extra work to transition to if you want to be compatible
> with older versions of Python that wouldn't have the header files (Victor's
> compatibility project could help here). The question for the team is
> whether separate header files makes sense to others, or would people prefer
> using `#define` and `Python.h` to control API access/opt-in?
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NOIKBK6MREMWLRLN76KZTHRABKKOHWUN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Steve Dower]

I prefer separate header files, provided people outside of core always 
have one (presumably "Python.h") that should be included first and 
includes enough info to check which headers will be available (i.e. the 
version defs).


Modifying preprocessor definitions for different Python versions, or 
having to set them before knowing what version is being used, seems more 
complicated.


Cheers,
Steve

On 5/30/2022 8:26 PM, Brett Cannon wrote:

We discussed having leading underscores for this API tier, and it was decided 
that a leading underscore was preferred.

This did start a discussion, though, about whether we should control API 
access/opt-in via `#include` by having `.h` files that convey what API the user 
is opting into, or use `#define` to control what gets exposed via `Python.h`. 
The general feeling was that the header file idea is ideal, but it is a little 
extra work to transition to if you want to be compatible with older versions of 
Python that wouldn't have the header files (Victor's compatibility project 
could help here). The question for the team is whether separate header files 
makes sense to others, or would people prefer using `#define` and `Python.h` to 
control API access/opt-in?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NMIULT7IJA77KYNFQGNQYV5LOVLV3FSV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Guido van Rossum]

I would love to see header files used for this -- while I know there is a
long tradition of feature-flags that must be #defined by the user before
#including a header in order to affect what the header exports (or not!),
30 years later I still find that approach pretty unintuitive.

But yes, it's going to be a complex transition.


On Mon, May 30, 2022 at 12:30 PM Brett Cannon  wrote:

> We discussed having leading underscores for this API tier, and it was
> decided that a leading underscore was preferred.
>
> This did start a discussion, though, about whether we should control API
> access/opt-in via `#include` by having `.h` files that convey what API the
> user is opting into, or use `#define` to control what gets exposed via
> `Python.h`. The general feeling was that the header file idea is ideal, but
> it is a little extra work to transition to if you want to be compatible
> with older versions of Python that wouldn't have the header files (Victor's
> compatibility project could help here). The question for the team is
> whether separate header files makes sense to others, or would people prefer
> using `#define` and `Python.h` to control API access/opt-in?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Q5JU5YKGX2U2UAAILDH45S5UGN6GLVXT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GNDXKI3XTUI3TOOQI44BO5BUL6ZWQMI4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 689 – Unstable C API tier (was: Semi-stable C API tier)

[Brett Cannon]

We discussed having leading underscores for this API tier, and it was decided 
that a leading underscore was preferred.

This did start a discussion, though, about whether we should control API 
access/opt-in via `#include` by having `.h` files that convey what API the user 
is opting into, or use `#define` to control what gets exposed via `Python.h`. 
The general feeling was that the header file idea is ideal, but it is a little 
extra work to transition to if you want to be compatible with older versions of 
Python that wouldn't have the header files (Victor's compatibility project 
could help here). The question for the team is whether separate header files 
makes sense to others, or would people prefer using `#define` and `Python.h` to 
control API access/opt-in?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q5JU5YKGX2U2UAAILDH45S5UGN6GLVXT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it possible to view tokenizer output?

[Jack]

Hi Pablo, could you clarify please? Is that on the main branch, or would 
you be willing to share the code?


On 30/05/2022 16:23, Pablo Galindo Salgado wrote:
There is no *public* one but there is a private one accesible from 
Python I added for testing purposes.


On Mon, 30 May 2022, 15:17 Victor Stinner,  wrote:

On Mon, May 30, 2022 at 1:40 AM Eric V. Smith 
wrote:
> python -m tokenize < file-to-parse.py
>
> See the comment at the top of tokenize.py. IIRC, it
re-implements the
> tokenizer, it does not call the one used for python code.

Ah right, I would be surprised that there would be a public Python API
to get the tokenizer output, since there is no public C API for that
:-)

I just removed  header file since it was never usable outside
Python C internals: there is no public C API to just run the tokenizer
and gets its output.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D3EMINLPE6TXEENR27LOVNN7TTYM2GNE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it possible to view tokenizer output?

[Pablo Galindo Salgado]

Is on the main branch but as I mentioned is **exclusively** for internal
consumption:

https://github.com/python/cpython/blob/8136606769661c103c46d142e52ec88803f6/Lib/tokenize.py#L685

On Mon, 30 May 2022 at 17:37, Jack  wrote:

> Hi Pablo, could you clarify please? Is that on the main branch, or would
> you be willing to share the code?
> On 30/05/2022 16:23, Pablo Galindo Salgado wrote:
>
> There is no *public* one but there is a private one accesible from Python
> I added for testing purposes.
>
> On Mon, 30 May 2022, 15:17 Victor Stinner,  wrote:
>
>> On Mon, May 30, 2022 at 1:40 AM Eric V. Smith  wrote:
>> > python -m tokenize < file-to-parse.py
>> >
>> > See the comment at the top of tokenize.py. IIRC, it re-implements the
>> > tokenizer, it does not call the one used for python code.
>>
>> Ah right, I would be surprised that there would be a public Python API
>> to get the tokenizer output, since there is no public C API for that
>> :-)
>>
>> I just removed  header file since it was never usable outside
>> Python C internals: there is no public C API to just run the tokenizer
>> and gets its output.
>>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UXPSZFOKCKGHUERUVO7UPLZK3L53CGFW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it possible to view tokenizer output?

[Pablo Galindo Salgado]

There is no *public* one but there is a private one accesible from Python I
added for testing purposes.

On Mon, 30 May 2022, 15:17 Victor Stinner,  wrote:

> On Mon, May 30, 2022 at 1:40 AM Eric V. Smith  wrote:
> > python -m tokenize < file-to-parse.py
> >
> > See the comment at the top of tokenize.py. IIRC, it re-implements the
> > tokenizer, it does not call the one used for python code.
>
> Ah right, I would be surprised that there would be a public Python API
> to get the tokenizer output, since there is no public C API for that
> :-)
>
> I just removed  header file since it was never usable outside
> Python C internals: there is no public C API to just run the tokenizer
> and gets its output.
>
> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5AWZ4NJCS5TVWMR2KZOWLJT7AYDCXVIY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it possible to view tokenizer output?

[Victor Stinner]

On Mon, May 30, 2022 at 1:40 AM Eric V. Smith  wrote:
> python -m tokenize < file-to-parse.py
>
> See the comment at the top of tokenize.py. IIRC, it re-implements the
> tokenizer, it does not call the one used for python code.

Ah right, I would be surprised that there would be a public Python API
to get the tokenizer output, since there is no public C API for that
:-)

I just removed  header file since it was never usable outside
Python C internals: there is no public C API to just run the tokenizer
and gets its output.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Raw strings ending with a backslash

[Steven D'Aprano]

Thank you to everyone who responded, it is now clear to me that this 
genuinely is a feature, not a bug or limitation of the parser or lexer. 
And that there is code relying on that behaviour, including in the 
stdlib, so we shouldn't change it even if we could.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RXA3TG5WN6YJ4HUVEUEOJWKYL27FPZVI/
Code of Conduct: http://python.org/psf/codeofconduct/