[Python-Dev] Re: Switching to Discourse
What happened to this SC decision (move to Discourse)? People started again to write on python-dev. So what's going on? Should I reply on python-dev? Ask to move to Discourse? Should we *close* the python-dev mailing list? Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6KOQZOOBJZU2X5CARMZ33RQBVEYIIKLC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Embedded Python Memory Leaks
Alexei is about about https://github.com/python/cpython/issues/96853 I suggest discussing the issue there. Victor On Thu, Oct 6, 2022 at 9:54 PM wrote: > > Hi Python team > > Is any plan to try and fix memory leaks when embedding Python in a C/C++ > application? > > Thanks in advance, > Alexei > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/32RBAGTWWJDAJJ3EWAXGMQOPYOXH3TB5/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FYF5MSMXQRFSYVJZFP3ZX277GGOQ7MFK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install
On Thu, Jun 16, 2022 at 7:54 PM Christian Heimes wrote:
> What Victor means that we would like to have clear instructions how to
> reproduce the problem ourselves. Could you please provide step by step
> instructions how I could set up a build environment on a X86_64 PC with
> a standard Linux distro (Fedora, Debian/Ubuntu)?
>
> What packages have to be installed? Do I have to download any extra
> packages? How do I have to set up my build environment? Which commands
> do I have to execute? Is there a container image available that comes
> with everything pre-installed?
I don't know if it helps, but I managed to build Python linked to the
musl libc using musl-gcc.
The build and the built Python just work on Fedora 36. It doesn't seem
to reproduce the discussed build issue.
Commands:
---
sudo dnf install musl-libc musl-gcc
git clean -fdx
./configure --with-pydebug CC=musl-gcc LD=musl-gcc
make
---
Interesting part of the build:
---
Platform "x86_64-pc-linux-musl" with compiler "gcc" is not supported by the
CPython core team, see https://peps.python.org/pep-0011/ for more information.
---
The executable is linked to ld-musl-x86_64.so.1:
---
$ ldd ./python
linux-vdso.so.1 (0x7fffbd6b6000)
ld-musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f5253669000)
---
Python works as expected:
---
$ ./python
Python 3.12.0a0 (heads/main:4beee0c7b0, Jun 17 2022, 12:23:58) [GCC
12.1.1 20220507 (Red Hat 12.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
2
---
The _struct extension works as expected and is linked to musl:
---
$ ./python
>>> import _struct
>>> _struct.pack('I', 4)
b'\x04\x00\x00\x00'
>>> _struct
$ ldd
/home/vstinner/python/main/build/lib.linux-x86_64-3.12-pydebug/_struct.cpython-312d-x86_64-linux-musl.so
linux-vdso.so.1 (0x7ffedb5fd000)
ld-musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f4be8cd)
---
There are some compiler warnings and build errors, but I didn't pay
attention to them since I was interested by the discussed build issue.
---
I also tried something closer to "cross-compiler", but I got it wrong,
since Python is still linked to the glibc, not to musl. Again, I don't
reproduce the issue.
By the way, what is the issue? Is it a build error?
bpo-39399 mentions the error "ImportError: libc.so: cannot open shared
object file" on "import struct".
Get Python 3.12 on the host (installed in /opt/py3.12):
---
git clean -fdx
./configure --prefix=/opt/py3.12
make
make install
---
Fake buildchain:
---
$ mkdir cross-build
$ cd cross-build/
$ ln -s /usr/bin/readelf x86_64-pc-linux-musl-readelf
$ cd ..
---
Build:
---
$ sudo dnf install musl-libc # install musl
$ cat config-musl
ac_cv_file__dev_ptmx=no
ac_cv_file__dev_ptc=no
$ cat build.sh
set -e -x
PATH=$PATH:cross-build/ \
CONFIG_SITE=config-musl \
./configure \
--with-build-python=/opt/py3.12/bin/python3.12 \
--build=x86_64-pc-linux-gnu \
--host=x86_64-pc-linux-musl \
--with-pydebug \
--disable-ipv6 \
--cache-file=../configure-musl.cache
$ ./build.sh
$ make
---
Sadly, the built executable is linked to the glibc:
---
$ ldd ./python
linux-vdso.so.1 (0x7ffcd89d)
libm.so.6 => /lib64/libm.so.6 (0x7f54d2a6)
libc.so.6 => /lib64/libc.so.6 (0x7f54d285f000)
/lib64/ld-linux-x86-64.so.2 (0x7f54d2b62000)
---
Well, since glibc and musl are available and just work on my system,
I'm not sure if my test makes any sense :-) Should the target libc not
work on the host to reproduce the issue?
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/3AKU5VJ5OBFZ7MVS4UWYLUAW5ZGGACIS/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install
On Thu, Jun 16, 2022 at 4:29 PM Xavier de Gaye wrote: > A Google search on "andoid x86_64" gives about 10,900,000 results, showing > that cross-compiling is quite common for the case where the build system and > the host system have the same PLATFORM_TRIPLET as when building android > x86_64 on an x86_64. So the argument that this problem is not worth fixing > because no one has to cross-compile in this environment, IMHO this argument > is not valid. You misunderstood me. I didn't say that the issue must not be fixed. I only tried to explain why it was not fixed yet. > Yes this issue does explain how to reproduce the problem. It even shows the > backtrace printed when attempting to cross-build Android x86_64 on a linux > x86_64. > To reproduce the issue, just cross-build python for android x86_64 following > the well documented process by the Android team. > It is not clear how this backtrace can be missed when reading the issue ! Is there a way to reproduce the issue without Android? Like building Python on x86-64 for musl on a system using glibc? > There is also a patch provided in this issue that is straightforward and that > does not involve any change on distutils, only the Makefile and configure. That's interesting, since distutils is scheduled from removal in Python 3.12. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B74YHEAKQZF2XZEOH6SKZRHXQGU4A5OD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Python-Help] Unable to bootstrap Python 3 install
Hi,
While this problem is causing you a lot of troubles, I never had to
cross compile Python, and I guess that it's the case for most people.
Changing the Python build system and distutils is stressful since it
can break Python for the majority of users, rather than leaving the
minority of users with an old bug. So far, nobody was brave enough to
"break" the system for cross compilation.
Moreover, as far as I know, cross compiling Python works for the
common case: different platform triplet. Only the corner case of the
same triple is causing troubles.
https://github.com/python/cpython/issues/66913 doesn't explain how to
reproduce the issue, it only gives some info about what doesn't work.
I don't know how to reproduce the issue. Please comment the issue.
To cross compile Python, I found these documentations:
* https://docs.python.org/dev/using/configure.html#cross-compiling-options
* WASM: https://github.com/python/cpython/blob/main/Tools/wasm/README.md
Currently, setup.py checks for:
CROSS_COMPILING = ("_PYTHON_HOST_PLATFORM" in os.environ)
Victor
On Tue, Jun 14, 2022 at 1:49 AM Dave Blanchard wrote:
> Here's what's happening. This is a very old problem reported ages ago which
> has never been fixed. If I set the target arch to i686 (on an x86_64 build
> system) Python will cross-compile and bootstrap just fine. But if the host
> and build triple are the same, then the problem will occur. Python seems to
> be ASSuming that if the build and host triple match (in this case, both being
> 'x86_64-linux-gnu') therefore the host and build libraries are
> binary-compatible--which is NOT actually the case when one is cross-compiling
> to a different libc, for example. Actually it's kind of brain dead how this
> is implemented.
>
> Some prior discussions/years-old bug reports:
>
> https://bugs.python.org/issue39399
> https://bugs.python.org/issue22724
> https://bugs.gentoo.org/705970
>
> In those links, various hacks are attempted with various degrees of success,
> but then Xavier de Gaye reworked the build system, apparently fixing the
> problem with this pull request in Dec. 2019:
>
> https://github.com/python/cpython/pull/17420
>
> Unfortunately he became annoyed in the comments, seemingly mostly due to the
> lack of interest from Python to actually do anything about this problem, and
> cancelled his pull request. His fix was never implemented, and Python
> cross-compiling remains broken to this day.
>
> I downloaded his patches, made a minor fix or two, and merged them all
> together into the one patch attached to this email. When applied to both my
> build system and target Python, this problem seems to be resolved, for me at
> least. I'll know more later tonight as I get further into the distro build
> process.
>
> It's surprising to hear that cross-compiling Python would be any kind of
> "unusual" thing, considering this is something that *always* has to be done
> any time one is bootstrapping anything on a new or different system. It
> surprises me further to see that Python actually requires the *minor* version
> number to be the same for bootstrapping, also. So not only is Python 3
> essentially a different language from Python 2, each point release is
> different and incompatible also. Amazing.
>
> I guess this shouldn't be surprising, after all of the other questionable
> design decisions this project has made over the years. I probably also
> shouldn't be surprised to see such an incredibly important bug going unfixed
> after all this time. It's Python--by far the #1 biggest annoyance and pain in
> the ass out of the 1,000+ packages on my distro, ranking just above CUPS and
> Texlive.
>
> Dave
>
>
> On Mon, 13 Jun 2022 16:12:26 -0500 (CDT)
> Matthew Dixon Cowles wrote:
>
> > Dear Dave,
> >
> > > Hello, I'm trying to cross compile a new version of my Linux system
> > > with some upgraded packages, including a new Glibc.
> >
> > I've never had to do that and I am beginning to suspect, from the
> > lack of replies here better than this one, that nobody else here has
> > had to either.
> >
> > > I've hit a major snag with Python 3.7 (also tried 3.9 with same
> > > result) which makes it through the compile OK, but then bombs when
> > > running ensurepip at the end.
> >
> > If it were me, I'd set --with-ensurepip to no, declare victory, and
> > run ensurepip on the target machine.
> >
> > I haven't been able to find any very good evidence, but I wouldn't be
> > surprised if the option to turn ensurepip off is there to help out
> > with cross-compiling. For example, here it's being turned off for
> > compiling to web assembly:
> >
> > https://t.ly/_ZE3
> >
> > (alternatively:
> >
> > https://github.com/python/cpython/commit/
> > 9deb83468c56c7484645e6e3a6d0183cd6a0afd7
> >
> > )
> >
> > which looks to me a lot like what you're doing.
> >
> > What seems to be happening is that in Lib/esurepip/__init__.py is
> > calling run_module() in Lib/runpy.py and that's calling
>
[Python-Dev] Re: Deadlock when interrupting interpreter initialisation with ptrace?
On Mon, Jun 6, 2022 at 4:35 PM Gabriele wrote: > The austinp variant is a variant of Austin > (https://github.com/P403n1x87/austin) for Linux that uses ptrace to > seize and interrupt/continue threads to capture native stack traces > using libunwind. During testing, I have discovered that there are good > chances of causing what looks like a deadlock in Python if the seizing > and interrupting of threads happen very early when spawning a Python > subprocess from austinp. Do you have a backtrace of the Python main thread when the hang happens? How do you spawn a new process? With the Python subprocess module? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3OUS6BAV3B7A4BQBCLHLKUQMJM3PF646/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 681 – Data Class Transforms
Hi, Side note: it would be nice to add "typing: " prefix or mention "type annotation" or "type check" in the title of PEPs which are about that. Just from the PEP title, it's hard *for me* to guess that it's about type annotations. Examples of other PEP titles which confused me: PEP 612 – Parameter Specification Variables PEP 645 – Allow writing optional types as x? PEP 646 – Variadic Generics PEP 647 – User-Defined Type Guards PEP 673 – Self Type PEP 675 - Arbitrary Literal String Type PEP 677 – Callable Type Syntax First, I understood that "Arbitrary Literal String Type" was adding a new built-in types for "literal strings" :-) Nope. It's just about type annotations ;-) From what I understood, the purpose of these PEPs outside type annotations is limited or non existent :-) Victor On Mon, Jun 6, 2022 at 2:51 PM Petr Viktorin wrote: > > Hello, > With the latest wording changes, PEP 681 – Data Class Transforms is now > fully accepted. Feel free to mark it as such at your convenience. > > Happy typing, > — Petr, on behalf of the Steering Council > > > On 23. 04. 22 13:26, Petr Viktorin wrote: > > Hello, > > As an initial implementation that will be improved in the future, the > > specification in PEP 681 is fine. Feel free to add the decorator to > > Python 3.11 at your convenience. > > > > However, the PEP includes several worrying recommendations like: > > > > - we recommend that the maintainers of attrs move away from the legacy > > semantics and adopt auto_attribs behaviors by default. > > - We chose not to support this feature and recommend that attrs users > > avoid converters. > > - Attrs users should use the dataclass-standard eq and order parameter > > names instead. > > > > These are probably meant as recommendations from typing-sig, but an > > accepted PEP represents consensus of the entire Python community. A > > typing PEP is not an appropriate place to make recommendations like > > this, especially without reaching out to the maintainer of attrs. > > As far as I know,the attrs and pydantic libraries are using the > > reference implementation, but their authors weren't consulted on the PEP > > itself. > > > > Could you either change the wording (e.g. say that the unsupported > > features need bespoke type-checker functionality for proper type > > checking), or work with attrs to make the same recommendations in its > > documentation? > > > > > > Happy typing, > > — Petr, on behalf of the Steering Council > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/R4A2IYLGFHKFDYJPSDA5NFJ6N7KRPJ6D/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LVNXFRLEMJXTEPC3N4M3NKQ4YRYQQZTA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Looking for volunteers to maintain the multiprocessing and concurrent.futures modules
Hi,
On buildbots, it's common that we get at least one multiprocessing
test failure per week. While I just reported a new one, I wanted to
add a "expert-multiprocessing" label, but it didn't exist. I just
created the label :-)
https://github.com/python/cpython/labels/expert-multiprocessing
I added the label manually to 240 issues! Oh, they are many open
issues about multiprocessing and concurrent.futures modules.
Documentation issues, feature requests, but also a long list of real
bugs ("hang" is a common term in these reports).
Handling multiple processes is hard to get it right, especially when
supporting multiple operating systems matter. In Python 3.8,
multiprocessing got shared memory support which opened another can of
worms :-)
Sadly, current multiprocessing maintainers are no longer available and
the number of open issues and open pull requests is increasing.
I'm not an expert in multiprocessing, but I'm volunteer to mentor
someone who is expert and is interested to go through these lists. I
cannot dedicate a lot of time, but I will try to be my best to be
available :-)
See also the GitHub Projects:
Multiprocessing issues:
https://github.com/orgs/python/projects/14
Pickle and copy issues:
https://github.com/orgs/python/projects/9
If you are interested, contact me in private. Sorry, I am not
available to teach you how multiprocessing works. I can only assist
people who already know it.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6FZECDUVJVEYY7DCCK2OQO66NIEQUE5M/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Is it possible to view tokenizer output?
You should maybe move the code out of the stdlib (to tests?) if it should not be used. Otherwise, someone somehow will start to rely on it, and then complain when it breaks :-) Victor On Mon, May 30, 2022 at 6:51 PM Pablo Galindo Salgado wrote: > > Is on the main branch but as I mentioned is **exclusively** for internal > consumption: > > https://github.com/python/cpython/blob/8136606769661c103c46d142e52ec88803f6/Lib/tokenize.py#L685 > > On Mon, 30 May 2022 at 17:37, Jack wrote: >> >> Hi Pablo, could you clarify please? Is that on the main branch, or would you >> be willing to share the code? >> >> On 30/05/2022 16:23, Pablo Galindo Salgado wrote: >> >> There is no *public* one but there is a private one accesible from Python I >> added for testing purposes. >> >> On Mon, 30 May 2022, 15:17 Victor Stinner, wrote: >>> >>> On Mon, May 30, 2022 at 1:40 AM Eric V. Smith wrote: >>> > python -m tokenize < file-to-parse.py >>> > >>> > See the comment at the top of tokenize.py. IIRC, it re-implements the >>> > tokenizer, it does not call the one used for python code. >>> >>> Ah right, I would be surprised that there would be a public Python API >>> to get the tokenizer output, since there is no public C API for that >>> :-) >>> >>> I just removed header file since it was never usable outside >>> Python C internals: there is no public C API to just run the tokenizer >>> and gets its output. >>> >>> Victor >>> ___ >>> Python-Dev mailing list -- python-dev@python.org >>> To unsubscribe send an email to python-dev-le...@python.org >>> https://mail.python.org/mailman3/lists/python-dev.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/ >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/UXPSZFOKCKGHUERUVO7UPLZK3L53CGFW/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R43LSIR3JN7FHVHO26T3SOLKTAG3J4DQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Is it possible to view tokenizer output?
On Mon, May 30, 2022 at 1:40 AM Eric V. Smith wrote: > python -m tokenize < file-to-parse.py > > See the comment at the top of tokenize.py. IIRC, it re-implements the > tokenizer, it does not call the one used for python code. Ah right, I would be surprised that there would be a public Python API to get the tokenizer output, since there is no public C API for that :-) I just removed header file since it was never usable outside Python C internals: there is no public C API to just run the tokenizer and gets its output. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CT3YSWSPMJ5DLUCVBX3AAPRWOUOXYWEL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: python3.110b1 fails during post compile setup on OpenBSD 7.1
Hi, Please report Python issues to the GitHub project: https://github.com/python/cpython/issues/ The OpenBSD operating system is currently not supported by Python, so don't expect a quick feedback: https://peps.python.org/pep-0011/ A few years ago, I tried to provide best effort support for OpenBSD. More recently, I focused on FreeBSD which already keeps me busy with multiple issues ;-) Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/COWYGGZQLTY2QH7OMZ4XKGFULGVSKWFY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: New public PyUnicodeBuilder C API
On Mon, May 16, 2022 at 2:11 PM Antoine Pitrou wrote: > > PyUnicodeBuilder_Init(); > > > > // Overallocation is more efficient if the final length is unknown > > PyUnicodeBuilder_EnableOverallocation(); > > PyUnicodeBuilder_WriteStr(, key); > > PyUnicodeBuilder_WriteChar(, '='); > > > > // Disable overallocation before the last write > > PyUnicodeBuilder_DisableOverallocation(); > > Having to manually enable or disable overallocation doesn't sound right. > Overallocation should be done *before* writing, not after. If there are > N bytes remaining and you write N bytes, then no reallocation should > occur. Calling these functions has no immediate effect on the current buffer. EnableOverallocation() doesn't enlarge the buffer. Even if the buffer is currently "over allocated", DisableOverallocation() leaves the buffer unchanged. Only the next writes will use a different strategy depending on the current setting. Only the Finish() function shrinks the buffer. Currently, it's the _PyUnicodeWriter.overallocate member. If possible, I would prefer to not expose the structure members in the public C API. Overallocation should be enabled before writing and disabled before the last write. It's disabled by default. For some use cases, it's more efficient to not enable overallocation (default). Always enabling overallocation makes the code less efficient. For example, a single write of 10 MB allocates 15 MB on Windows and then shinks the final string to 10 MB. Note: The current _PyUnicodeWriter implementation also has an optimization when there is exactly one single WriteStr(obj) operation, Finish() returns the input string object unchanged, even if overallocation is enabled. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/POMWWUW6DH2Y3OZOGAHPIFX4JDFYQ2SK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: New public PyUnicodeBuilder C API
On Mon, May 16, 2022 at 12:51 PM wrote: > > Victor Stinner wrote: > > On Mon, May 16, 2022 at 11:40 AM dw-...@d-woods.co.uk wrote: > > > Cython used the private _PyUnicodeWriter API (and stopped using it on > > > Py3.11 when it was hidden more thoroughly) > > > I'm not aware of any change in the the private _PyUnicodeWriter API in > > Python 3.11. > > It was _PyFloat_FormatAdvancedWriter and _PyLong_FormatAdvancedWriter that > got moved internally to somewhere Cython couldn't easily get them I think. > (https://github.com/python/cpython/commit/0a883a76cda8205023c52211968bcf87bd47fd6e > and > https://github.com/python/cpython/commit/5f09bb021a2862ba89c3ecb53e7e6e95a9e07e1d). > Obviously it would be possible to include the internal headers and re-enable > it though - just turning it off was the quickest way to get it working at the > time though I moved these "advanced formatter" functions to the internal C API in batch of changes which moved most private functions to the internal C API. If you consider that they are useful outside Python, please open an issue to request expose them as public functions. Right now, the problem is that they use the _PyUnicodeWriter API which is also private. If a public API is added to "build a string", maybe it would make sense to add these "advanced formatter" functions to the public C API? My proposed API targets Python 3.12, it's too late for Python 3.11. Maybe for Python 3.11, it's ok to add back private _PyFloat_FormatAdvancedWriter and _PyLong_FormatAdvancedWriter functions to the public C API to restore Cython performance. Sadly, Cython still has to be changed at each Python release because it still uses many private functions and private functions change often. We have to go through this process to think about these APIs and decide which ones should become public C functions, and which ones are fine to be fully internal. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OASHJSB5HI2VN3RBCV5T3CYFTP4TZYOC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: New public PyUnicodeBuilder C API
On Mon, May 16, 2022 at 11:40 AM wrote: > Cython used the private _PyUnicodeWriter API (and stopped using it on Py3.11 > when it was hidden more thoroughly) I'm not aware of any change in the the private _PyUnicodeWriter API in Python 3.11. Is it just that Cython no longer wants to use private APIs? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4HIQ2O3BNABK5XA5DW5HSLJELMO3XARP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] New public PyUnicodeBuilder C API
Hi,
I propose adding a new C API to "build an Unicode string". What do you
think? Would it be efficient with any possible Unicode string storage
and any Python implementation?
PyPy has an UnicodeBuilder type in Python, but here I only propose C
API. Later, if needed, it would be easy to add a Python API for it.
PyPy has UnicodeBuilder to replace "str += str" pattern which is
inefficient in PyPy: CPython has a micro-optimization (in ceval.c) to
keep this pattern performance interesting. Adding a Python API was
discussed in 2020, see the LWN article:
https://lwn.net/Articles/816415/
Example without error handling, naive implementation which doesn't use
known length of key and value strings (calling Preallocate may be more
efficient):
---
// Format "key=value"
PyObject *format_with_builder(PyObject *key, PyObject *value)
{
assert(PyUnicode_Check(key));
assert(PyUnicode_Check(value));
// Allocated on the stack
PyUnicodeBuilder builder;
PyUnicodeBuilder_Init();
// Overallocation is more efficient if the final length is unknown
PyUnicodeBuilder_EnableOverallocation();
PyUnicodeBuilder_WriteStr(, key);
PyUnicodeBuilder_WriteChar(, '=');
// Disable overallocation before the last write
PyUnicodeBuilder_DisableOverallocation();
PyUnicodeBuilder_WriteStr(, value);
PyUnicode *str = PyUnicodeBuilder_Finish();
// ... use str ...
return
error:
PyUnicodeBuilder_Clear();
...
}
---
Proposed API (11 functions, 1 type):
---
typedef struct { ... } PyUnicodeBuilder;
void PyUnicodeBuilder_Init(PyUnicodeBuilder *builder);
int PyUnicodeBuilder_Preallocate(PyUnicodeBuilder *builder,
Py_ssize_t length, uint32_t maxchar);
void PyUnicodeBuilder_EnableOverallocation(PyUnicodeBuilder *builder);
void PyUnicodeBuilder_DisableOverallocation(PyUnicodeBuilder *builder);
int PyUnicodeBuilder_WriteChar(PyUnicodeBuilder *builder, uint32_t ch);
int PyUnicodeBuilder_WriteStr(PyUnicodeBuilder *builder, PyObject *str);
int PyUnicodeBuilder_WriteSubstr(PyUnicodeBuilder *builder,
PyObject *str, Py_ssize_t start, Py_ssize_t end);
int PyUnicodeBuilder_WriteASCIIStr(PyUnicodeBuilder *builder,
const char *str, Py_ssize_t len);
int PyUnicodeBuilder_WriteLatin1Str(PyUnicodeBuilder *builder,
const char *str, Py_ssize_t len);
PyObject* PyUnicodeBuilder_Finish(PyUnicodeBuilder *builder);
void PyUnicodeBuilder_Clear(PyUnicodeBuilder *builder);
---
The proposed API is based on the private _PyUnicodeWriter C API that I
added in Python 3.3 to optimize PEP 393 implementation.
PyUnicodeBuilder_WriteASCIIStr() is an optimization: in release mode,
the function doesn't have to check if the string contains non-ASCII
characters. In debug mode, it must fail. If you consider that this API
is too likely to introduce bugs in release mode, it can be removed.
PyUnicodeBuilder_Preallocate() maxchar can be zero, but for the
current Python implementation (PEP 393 compact string), it's more
efficient if maxchar matchs the expected Unicode storage: 127 for
ASCII, 255 for Latin1, 0x for UCS2, or 0x10 for UCS4. The
value doesn't have to the exact, for example, it can be any valiue in
[128; 255] for Latin1. The problem is that computing maxchar (need to
read characters, decode a byte strings from a codec, etc.) can be
expensive and an PyUnicodeBuilder_Preallocate() implementation can
ignore maxchar depending on the chosen Unicode string storage.
PyUnicode_MAX_CHAR_VALUE(str) can be used to create maxchar, but this
function is specific to the current CPython implementation.
Maybe a second "preallocate" function without maxchar should be added
(more convenient, but less efficient). I don't know.
Rationale for adding a new public C API.
Currently, the Python C API is designed to allocate an Unicode string
on the heap memory with uninitialized characters, and then basically
gives a direct access to these characters. Since Python 3.3, creating
an Unicode string in a C extension became more complicated: the caller
must know in advance what will be the optimal storage for characters:
ASCII, Latin1 UCS1 [U+; U+00FF], BMP UCS-2 [U+; U+],or
full Unicode character set UCS4 [U+; U+10]. When writing a
codec decoder (like decoding UTF-8), the maximum code point is not
known in advance and so the decoder may need to change the buffer
format while decoding (start with UCS1, switch to UCS2, maybe switch a
second time to UCS4).
The current C API has multiple drawbacks:
* It is designed to target the exact format "PEP 393 compact strings"
("kind + data").
* It is inefficient for PyPy which uses UTF-8 internally. So it would
also be inefficient if Python is modified to also use UTF-8
internally.
* It leaks too many implementation details.
* It creates an uninitialized string which might
[Python-Dev] Re: Summary of Python tracker Issues
According to my notes, the summary is generated by this Python script:
https://github.com/psf/bpo-tracker-cpython/blob/master/scripts/roundup-summary
But I don't know what runs this script once a week.
Victor
On Fri, May 13, 2022 at 8:56 PM Brett Cannon wrote:
>
> Can we shut this down or unsubscribe the weekly email?
>
> On Fri, May 13, 2022 at 11:13 AM Python tracker
> wrote:
>>
>>
>> ACTIVITY SUMMARY (2022-05-06 - 2022-05-13)
>> Python tracker at https://bugs.python.org/
>>
>> To view or respond to any of the issues listed below, click on the issue.
>> Do NOT respond to this message.
>>
>> Issues counts and deltas:
>> open7146 ( +0)
>> closed 51841 ( +0)
>> total 58987 ( +0)
>>
>> Open issues with patches: 2890
>>
>>
>> Most recent 15 issues with no replies (15)
>> ==
>>
>> #47258: Python 3.10 hang at exit in drop_gil() (due to resource warnin
>> https://bugs.python.org/issue47258
>>
>> #47256: re: limit the maximum capturing group to 1,073,741,823, reduce
>> https://bugs.python.org/issue47256
>>
>> #47253: LOAD_GLOBAL instruction with wrong source position
>> https://bugs.python.org/issue47253
>>
>> #47252: socket.makefile documentation is missing data regarding the 'b
>> https://bugs.python.org/issue47252
>>
>> #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE
>> https://bugs.python.org/issue47251
>>
>> #47244: email.utils.formataddr does not respect double spaces
>> https://bugs.python.org/issue47244
>>
>> #47242: Annoying white bar in IDLE (line 457 in sidebar.py)
>> https://bugs.python.org/issue47242
>>
>> #47241: [C API] Move the PyCodeObject structure to the internal C API
>> https://bugs.python.org/issue47241
>>
>> #47238: Python threading.Event().wait() depends on the system time
>> https://bugs.python.org/issue47238
>>
>> #47236: Document types.CodeType.replace() changes about co_exceptionta
>> https://bugs.python.org/issue47236
>>
>> #47228: Document that na??ve datetime objects represent local time
>> https://bugs.python.org/issue47228
>>
>> #47222: subprocess.Popen() should allow capturing output and sending i
>> https://bugs.python.org/issue47222
>>
>> #47219: asyncio with two interpreter instances
>> https://bugs.python.org/issue47219
>>
>> #47218: adding name to lzmafile
>> https://bugs.python.org/issue47218
>>
>> #47217: adding name to BZ2File
>> https://bugs.python.org/issue47217
>>
>>
>>
>> Most recent 15 issues waiting for review (15)
>> =
>>
>> #47256: re: limit the maximum capturing group to 1,073,741,823, reduce
>> https://bugs.python.org/issue47256
>>
>> #47255: Many broken :meth: roles in the docs
>> https://bugs.python.org/issue47255
>>
>> #47254: enhanced dir?
>> https://bugs.python.org/issue47254
>>
>> #47251: Merge BINARY_SUBSCR_LIST_INT with BINARY_SUBSCR_LIST_TUPLE
>> https://bugs.python.org/issue47251
>>
>> #47243: Duplicate entry in 'Objects/unicodetype_db.h'
>> https://bugs.python.org/issue47243
>>
>> #47233: show_caches option affects code positions reported by dis.get_
>> https://bugs.python.org/issue47233
>>
>> #47222: subprocess.Popen() should allow capturing output and sending i
>> https://bugs.python.org/issue47222
>>
>> #47218: adding name to lzmafile
>> https://bugs.python.org/issue47218
>>
>> #47217: adding name to BZ2File
>> https://bugs.python.org/issue47217
>>
>> #47216: adding mtime option to gzip open()
>> https://bugs.python.org/issue47216
>>
>> #47215: Add "unstable" frame stack api
>> https://bugs.python.org/issue47215
>>
>> #47208: Support libffi implementations that cannot support invocations
>> https://bugs.python.org/issue47208
>>
>> #47205: posix.sched_{get|set}affinity(-1) no longer returns ProcessLoo
>> https://bugs.python.org/issue47205
>>
>> #47200: Add ZipInfo.mode property
>> https://bugs.python.org/issue47200
>>
>> #47199: multiprocessing: micro-optimize Connection.send_bytes() method
>> https://bugs.python.org/issue47199
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/LEYLS2TYSZ4NVDFLTDSQUT25C2Y4QG2O/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/C53WJXVWHZNGLTTPFQXHUWIZCYF2FBP7/
> Code of Conduct: http://python.org/psf/codeofconduct/
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe
[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12
On Tue, May 10, 2022 at 7:18 PM Guido van Rossum wrote: > Shouldn't we wean our internal tools off this obsolete version of distutils > too, rather than just move it around? Here is a first PR to build the test C++ extension with setuptools in test_cppext: https://github.com/python/cpython/pull/92639 It implements what I proposed in the issue. It runs the test in a venv. Python stdlib (ensurepip) includes a setuptools wheel package and so is able to install it without adding any new external dependency. If this approach works, we can use the same for test_peg_generator. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S6UKSWFTJGPSTK2MZ4RIPVQJ7CKAJCQG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12
On Tue, May 10, 2022 at 6:16 PM Steve Dower wrote: > I agree. The internal tools that use it are all in our Tools directory, > so we can move distutils there and explicitly add an import path to > locate it. No need to keep it in the stdlib (Lib/) at all. > > Migrating to Makefile builds is probably better long-term, but not as > important as moving distutils out from the stdlib so that setuptools can > rely on their copy being the "main" one. With my current PR, Lib/_distutils/ is not installed by "make install". Moving it to Tools/ would also work. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T43OZ4LWUT7QBENWPKIA3LFT52GWJV3Q/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12
On Tue, May 10, 2022 at 5:33 PM Christian Heimes wrote: > We can easily take care of test_cppext and add the build step to > Makefile. What matters in test_cppext is that using the Python C API in C++ does not emit compiler warnings, so it uses -Werror. Since this test is very new, I would prefer to not block the Python build just because of that. Moreover, I'm not sure that I want to require a C++ compiler to build CPython (written in C). But yeah, something can be done in Makefile :-) > How does test_peg_generator depend on distutils? I don't see > an import of distutils in the directory. See Tools/peg_generator/pegen/build.py file. The test builds C extensions and then execute them. > I would prefer to fix and remove all imports of distutils before we > resort to renaming the package. Please give us time until alpha 1. There > is no need to rush it *now*. Sure, there is no need to rush it now. But I prefer to discuss the change as soon as possible (in the 3.12 dev cycle), to make sure that we are aware of all issues, and have more time to fix them. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L6ECTYWEHYITDBFVMY4HTYCVFXUD6W3S/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Migration plan for the distutils removal in Python 3.12
On Tue, May 10, 2022 at 11:28 AM Christian Heimes wrote: > > Right now, Python still uses distutils internally for multiple use > > cases. I propose to start with renaming the distutils package to > > _distutils in the stdlib: > > > > * https://github.com/python/cpython/issues/92584 > > * https://github.com/python/cpython/pull/92585 > > Erlend and I got most extension modules ported to autoconf and > Modules/Setup.stdlib. The remaining modules should be done in a couple > of weeks. I recommend that we do not rename distutils and instead remove > it entirely. test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils. There is also the c-analyzer tool which uses distutils to run a C preprocessor. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TFBMHY4JV3QJL5I7VB65G7VHVUMJD4GX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Migration plan for the distutils removal in Python 3.12
Hi, Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions. * Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal? * Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools? * Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system. * Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package. * Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues? A code search in top 5000 PyPI projects (at 2022-01-26) using the regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects. Some statistics: * 851 projects (1,372 lines) import distutils in setup.py * 233 projects (700 lines) use distutils.version * 92 projects (205 lines) use distutils.util * 1,018 lines are type annotations (.pyi files) I failed to group the remaining 2,272 lines. Converting 851 projects from setup.py with distutils to pyproject.toml is going to take a few months if not years. Python 3.12 will likely be released in October 2023 (in year and a half). Since setuptools 60 (December 2021), setuptools installs a distutils-precedence.pth file to override the stdlib module with its local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib environment variable can be used to explicitly request the stdlib flavor. If I understood correct, distutils is now maintained in setuptools, rather than in the Python stdlib. Python "make install" installs an up to date setuptools and so indirectly its distutils-precedence.pth file which makes the distutils local copy of setuptools available. But many Linux distributions splits setuptools and pip from the "python" package. Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib: * https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585 Right now, the change is blocked by pip which still imports distutils (even if it prefers sysconfig in practice on Python 3.10 and newer). I reported the issue to pip, it should be easy to fix. Once Python will no longer depend on _distutils, it will be possible to fully remove it. By the way, Fedora 35 now longer provides automatically setuptools when building a Python package. A dependency to setuptools must now be explicit, since they are projects which don't use setuptools nor distutils, but other build systems. https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setuptools Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XFZJ43VSIWCYNQ5ZYT64XF7PMH4YDUWC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Test Python 3.11 beta1 with PYTHONSAFEPATH=1
Hi, I added the -P command line option and the PYTHONSAFEPATH=1 environment variable to Python 3.11 beta 1 to not prepend an "unsafe path" to sys.path such as the script's directory or the current directory: https://docs.python.org/dev/using/cmdline.html#cmdoption-P Example: -- $ echo 'Nope!' >random.py # broken module $ echo 'import random; print(random.randint(1, 6))' >dice.py $ python3.11 dice.py # default behavior (...) File "/home/vstinner/random.py", line 1 (...) SyntaxError: invalid syntax $ python3.11 -P dice.py # ignore local random.py 4 -- Please test Python 3.11 beta1 with the PYTHONSAFEPATH=1 environment variable set, or at least run python with the -P option. I'm curious which use cases are not affected and which use cases are affected. The PYTHONSAFEPATH=1 environment variable is inherited and so affect child processes. It can break applications relying on Python 3.10 sys.path behavior. I proposed adding -p option which does the opposite of the -P option: opt-in for Python 3.10 "unsafe" sys.path behavior, ignore the PYTHONSAFEPATH environment variable: https://github.com/python/cpython/pull/92361 If PYTHONSAFEPATH=1 only breaks a minority of use cases, maybe we can consider to make it the default, as Perl did in Perl 5.26 (2017) to increase its security: https://perldoc.perl.org/perl5260delta#Removal-of-the-current-directory-(%22.%22)-from-@INC Perl has an environment variable to get the old (Perl 5.24) behavior: PERL_USE_UNSAFE_INC=1. If enough people consider that it would be a good idea to change the default, I can maybe write a full PEP (I already have some notes). Even if the PEP is rejected, it might be a good thing to write down everything about this topic since it's a common issue hit by users learning Python and a common question of people auditing the Python security. I was asked a few months ago about changing the default to increase Python security. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VO5A6K5O5HB563PNYZZ3SN57LIXRJTPC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [python-committers] Release of Python 3.11 beta 1 is currently blocked
On Fri, May 6, 2022 at 2:14 PM Pablo Galindo Salgado wrote: > Today we need to start the release of Python 3.11 beta 1. Currently, we have > the following blockers: > > * https://github.com/python/cpython/issues/91162 (was deferred blocker - but > all deferred blockers are bumped to release blockers on beta1). "PEP 646: Decide on substitution behavior" where PEP 646 is "PEP 646 – Variadic Generics", so something about typing. Who can take this decision? > * https://github.com/python/cpython/issues/91350 The main branch got a fix. The backport to 3.8 was missing because the Docs CI failed: I repaired the PR #32250 which is now waiting for Lukasz (Python 3.8 release manager) to merge it. > * We have 3 buildbots failing: > https://buildbot.python.org/all/#/release_status (some of the failures are > due to flaky tests but others look real). The FreeBSD buildbot has no working C compiler: I contacted Koobs who maintains it. It's unrelated to Python itself. TraceRefs: I wrote https://github.com/python/cpython/pull/92396 to fix it => merged A SSL test of test_asyncio failed on x86 Gentoo Non-Debug with X 3.x: this buildbot is very slow, and the timeout is hardcoded to 30 seconds. I proposed https://github.com/python/cpython/pull/92402 to use a timeout of 5 minutes (or longer, it's adapted to buildbot --timeout argument). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V6MVCOL5S7TWJA46INVNZGI7OCGT27MK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
Ah, I see that I didn't explain something well. The issue has two sides: one side is a fix security vulnerability, the second side is more about Python *usability*. The Python usability issue is that running "math.py" overrides the Python stdlib module called "math". "math.py" is just an example, Python 3.11 contains has 305 modules: a name conflict is likely, especially by users learning Python fall into this trap (create a file called "random.py" to play with the "random" module). The issue so common that IPython added a "launcher" to work around this issue, to remove sys.path[0]: https://github.com/ipython/ipykernel/commit/3f7a03d356eee3500261acf9eec6bad2196c2097 Another example is the pytest project: running "pytest [...]" is different than "python -m pytest [...]". The second command adds the current working directory which can change the test behavior. It's a documented issue: https://docs.pytest.org/en/latest/how-to/usage.html#calling-pytest-through-python-m-pytest For the pytest use case, you still want to add the user site directory to sys.path and you still want to accept PYTHON environment variables like PYTHONWARNINGS. The only thing that you don't want is to add the current working directory to sys.path. Read also this discussion by Miro Hrončok about this usability issue in Fedora: https://discuss.python.org/t/python-flag-envvar-to-not-put-current-directory-on-sys-path-but-dont-ignore-pythonpath/4235 Victor On Wed, Apr 27, 2022 at 5:57 PM Victor Stinner wrote: > > The use case for -P still uses environment variables like > PYTHONWARNINGS or PYTHONUTF8. That's why -I (isolated) cannot be used. > > If there is an use case for a ._pth file importing the site module, > maybe a different option can be added, no? Adding -P doesn't prevent > that. But it seems like use cases are different enough to justify two > options, no? > > Victor > > > On Tue, Apr 26, 2022 at 11:52 PM Victor Stinner wrote: > > The only purpose of proposed -P option is to "not add sys.path[0]". > > There are use cases which only need that. > > > > Victor > > > > On Tue, Apr 26, 2022 at 8:37 PM Steve Dower wrote: > > > > > > On 4/26/2022 10:46 AM, Victor Stinner wrote: > > > > I propose adding a -P option to Python command line interface to "not > > > > add sys.path[0]": > > > > https://github.com/python/cpython/pull/31542 > > > > > > > > See the documentation in the PR for the exact behavior of this option. > > > > I prefer to add an environment variable, only pass the option > > > > explicitly on the command line. > > > > > > Another viable option might be to add an option to imply "import site", > > > which would work together with -I to: > > > * ignore environment variables (-E/-I) > > > * omit implicit CWD imports (-I) > > > * still process .pth files (-?) > > > * still include site-packages and user site-packages in sys.path (-?) > > > > > > It seems likely that the proposed -P would almost always be used with > > > -E, since if you can't control CWD then you presumably can't control > > > environment variables either. > > > > > > The existing ._pth functionality starts by implying -I, and allows > > > "import site" in the file to explicitly include site. A command-line > > > option matching this behaviour would be consistent. There's also already > > > configuration in our structures for import site, so there'd be no need > > > to add new fields to public APIs for the option. > > > > > > The biggest issue I see is that the obvious command line options for > > > "import site" are already used to imply "do not import site". But then, > > > -P isn't obvious either. Maybe an -X option would suffice? > > > > > > Cheers, > > > Steve > > > > > > > > > -- > > Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/A65CZFDNPXGBMENQD56VHF2QXDS3ENIC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Can I ask a real dumb procedural question about GitHub email?
Every day, I unsubscribe from many issues and pull requests to reduce the number of emails to move it below my acceptable threshold. For example, for me, 20 new discussions per day is acceptable, whereas 100 is not. I prefer to focus on small number of things and follow them closely. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2E5D7WGNZL2BGQEQEURSJ4HN7WWTADLT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
Hi, I updated my PR https://github.com/python/cpython/pull/31542 and I plan to merge it soon. It seems like most people need and like this feature. About the feature name, nobody liked the "add_path0" name which is misleading: "path0" is not easy to get, and the path is prepended, not added. I renamed "add_path0" to "safe_path" (opposite meaning) and I renamed "PYTHONDONTADDPATH0" env var to "PYTHONSAFEPATH": shorter, easy to write. In terms of usability, IMO "safe_path=1" is easier to understand than "add_path0=0". For the exact meaning of this option, well, I wrote documentation. In the documentation, I replaced "don't add sys.path[0]" with "don't prepend an unsafe path to sys.path: (...)" with an explanation of which "unsafe path" is prepended. Adding this -P command line option makes the Python command line even more complicated, with existing -I and -E options, the "._pth" file, etc. But well, not all users want the same thing ;-) Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5HOOBV7AOH2P7CR4RA2IAESFVQ2OQHLL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Hi Antoine, I'm in favor of making the C API more strict. The limited C API of Python 3.11 no longer cast arguments to PyObject* ! The caller must now cast to PyObject*: see PEP 670 "convert macros to functions". Converting macros to static inline functions (PEP 670) is already an important change. I prefer to not change too many things at the same time. Maybe in Python 3.12 (or later), we can consider requiring to not pass "const PyObject*" in C++: remove the cast dropping constness of the argument. My long term goal is to only use opaque "PyObject*" "handles" in the C API. But for practical reasons, changing the C API must be done slowly, step by step. To reduce the maintenance burden in third party C and C+ extensions. -- IMO it's a bad idea to use "const PyObject*" in the C API. Maybe today, a function doesn't modify the memory. But maybe tomorrow, the implementation will deeply change and will need to modify the memory. In C, using "const" (for something other than const char*) is just not convenient. Victor On Tue, May 3, 2022 at 9:22 AM Antoine Pitrou wrote: > On Mon, 2 May 2022 15:21:24 +0200 > Victor Stinner wrote: > > > > Slowly, more and more tests can be added. For example, this change fixing > > compiler warnings when passing "const PyObject*" to Python C API > functions > > adds tests to _testcppext.cpp: > > https://github.com/python/cpython/pull/92138 > > Doesn't passing "const PyObject*" leak implementation details, for > example that the reference count does not change? It seems to go > counter the objective of making the C API more abstract and more stable. > > (C++ has the "mutable" keyword for such situat but I don't think C has > it yet) > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/YFD3WOOR6PRORTTKLTSSOSDNHRYAA2N7/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S3B2KMUL4OM5CCLTYCFCR5HFZBBFWVKV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Hi, I merged the best basic tests to make sure that using the Python C API in C++ does not emit compiler warnings: * Code: https://github.com/python/cpython/blob/main/Lib/test/_testcppext.cpp * Test: https://github.com/python/cpython/blob/main/Lib/test/test_cppext.py The code is only built to check for compiler warnings. Later, I plan to build this C++ extension to be able to *execute* it at runtime. For now, a C++ compiler is required to run the Python 3.11 test suite. Maybe it should be made optional, but so far I failed to test if distutils has an available C++ compiler. Slowly, more and more tests can be added. For example, this change fixing compiler warnings when passing "const PyObject*" to Python C API functions adds tests to _testcppext.cpp: https://github.com/python/cpython/pull/92138 Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5I6EJNRWDEHYL773TQHZWHNHG2ZAY3SI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Summary of Python tracker Issues
On Fri, Apr 29, 2022 at 8:12 PM Python tracker wrote: > ACTIVITY SUMMARY (2022-04-22 - 2022-04-29) > Python tracker at https://bugs.python.org/ > > To view or respond to any of the issues listed below, click on the issue. > Do NOT respond to this message. > > Issues counts and deltas: > open7146 ( +0) > closed 51841 ( +0) > total 58987 ( +0) Congrats! We finally managed to stop users from reporting new bugs. Python reached perfection Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/X4NPUP6OX2AS2E3EWBSAFAO543B7KTYW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 689 – Semi-stable C API tier
I think that the main advantage of "unstable" over "semi-stable" is that it's a single word :-D It avoids the really hard question (!) about the separator between "semi" and "stable" ;-) (semistable? semi-stable? semi_stable?). Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V76VFFMUZFNGSQBOEYDKJH22D6NFZJZ2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 689 – Semi-stable C API tier
> Rejected Ideas > == > > It might be good to add a similar tier in the Python (not C) API, > e.g. for ``types.CodeType``. > However, the opt-in mechanism would need to be different (if any). > This is outside the scope of the PEP. For types.CodeType constructor, would it be possible to just a mention in the *documentation* that this API is "unstable"? It would come with a link to definition of the "unstable" C API: explain that it can change in 3.x.y bugfix releases, not not in 3.x.0 releases (major? minor? I never recall how they should be called). For now, I don't think that there is a need to actively remove this API from the "default" Python API and add an opt-in option to get access to these functions. But having a mention just in the documentation would be better than nothing. It seems to be popular complain and request. For example, most of the ast module would fall into this "unstable API". Previous discussions: * Proposal: declare "unstable APIs" https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNMDAKXYD5O54QWMVR2X7QOXVL/ * Making code object APIs unstable https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/ On one side, it's important to communicate that the API *can* change in 3.x.0 releases, but also provide some warranties that the API *must not change* in 3.x.y bugfix releases. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AS7KUFPSGNRNBUKMXPHHKHXAVDAZH2AT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 689 – Semi-stable C API tier
Ok, let me start with the serious business: API name. I'm not comfortable with "semi-stable". Python already has a "limited API" and a "stable ABI". Just by its name, it's unclear what "semi-stable" means. Honestly, I would be more comfortable with the name: "unstable API". It would be clear that the API *can* change often. People who want to know exactly the backward compatibility warranties can dig into the API documentation to learn more about it. "Unstable API" is also the name the Guido proposed for PyCode_New() last year: * Proposal: declare "unstable APIs" https://mail.python.org/archives/list/python-dev@python.org/thread/JM6SQ2YNMDAKXYD5O54QWMVR2X7QOXVL/ * Making code object APIs unstable https://mail.python.org/archives/list/python-dev@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/ Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UIUCHEVHLUJJFDZASBDTUDHLW4PIZWLP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Hi, The C++ version was discussed in the 2nd link that I gave in my first message: https://github.com/python/cpython/issues/91321 Gregory wrote "If we can conditionally test new things based on C++XX version, accumulating modern issue regression tests seems useful. Otherwise 11 at minimum." Another data point is that I mentioned that pybind11 is an important project for Python ecosystem. This project name and its definition refer to C++11: "pybind11: Seamless operability between C++11 and Python". So far, the only C++ code used in Python is the very recent change that I made to use reinterpret_cast<> and static_cast<> in the Python C API (.h header files). The C code base of Python is, well, written only in C. I mentioned nullptr if we get compiler warnings on the NULL constant in static inline functions. nullptr was added by C++11. I also mentioned C++20 "module" keyword and explained that it doesn't affect the Python C API. Having believes and assumptions about C++ compatibility is one thing. My plan is more about actually *testing it* ;-) Victor On Fri, Apr 29, 2022 at 12:36 AM wrote: > > > Not for me to answer, I'm not a proponent of the change. I'm sure if > > you read past discussions here and on Discourse you'll find answers > > from the people who studied the problem carefully. > > The opening mail proposed C++11 without rationale or references. I did search > the archives and discourse before, but nothing stood out, and I don't think an > encyclopedic knowledge of past python-dev discussions is a reasonable > requirement to comment or propose a variation on its merits. > > > I thought you might have something to add to the conversation, but I guess > > not? > > I find this tone quite out-of-place. I made a proposal (based on compiler > support, > version hygiene and compatibility with newer standards), and I'd have been > more > than happy to hear arguments (like Antoine's) or references for the merits of > preferring C++11 (though, again, the point became moot since Victor correctly > pointed out we can test against several versions). > > Still, the insinuation (as it arrives on my end) that I shouldn't participate > seems really unnecessary. > > Best, > H. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/7SSI53EDU2U565O2TYRTU4CPYLVXPO5K/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HV6AV527HVU6REWNFABYFLNV6CHGJ6RN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Recently, a issue about C++20 compatibility was reported: "The Python library will not compile with a C++2020 compiler because the code uses the reserved “module” keyword" https://github.com/python/cpython/issues/83536 In fact, after a long study, Python C API is *not* affected by this issue. Using "module" remains valid in C++20: see the issue for details. Victor On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou wrote: > > On Thu, 28 Apr 2022 22:03:25 +0900 > "Stephen J. Turnbull" wrote: > > h.vetin...@gmx.com writes: > > > > > While I don't know who proposed C++11 or where, I'd therefore like > > > to propose to move to _at least_ C++14. > > > > What benefits does this have for Python development? > > Let me second that question as well. > > I work on Apache Arrow, where the C++ parts require C++11 (and we can't > go further than this for now because of R compatibility concerns). We > could say that enabling the Python bindings switches the required C++ > version to C++14, but that would bring complication for no actual again > given that you're not likely to benefit from C++14 features in the > header files of a *C* project, are you? > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AS2CRKHCPLXJ74CTMNURAZ5ANAKUNZ3J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Since we are you talking about tests, we can easily run the tests on multiple C++ versions. But we have to start somewhere, so I propose to start with C++11. More C++ versions can be tested later. Victor On Thu, Apr 28, 2022 at 5:54 AM wrote: > > > In terms of C++ version, it was proposed to target C++11. > > GCC 5 has full C++14 support (one library functionality missing), and so does > VS2015 onwards as well as Clang 3.4, see > https://en.cppreference.com/w/cpp/compiler_support > > I doubt that any older compilers are in use _anywhere_ in reasonable numbers > that this should constrain the development of CPython. > > While I don't know who proposed C++11 or where, I'd therefore like to propose > to move to _at least_ C++14. > > Note that https://github.com/python/peps/pull/2309 already bumped the > required C-standard to C11, and originally defined this as > > The C11 subset are features supported by GCC 8.5, > > clang 8.0, and MSVC of Visual Studio 2017. > > If those versions should be regarded as the lower bounds of compiler support > (are they - or anything lower - tested on the build bots...?), then C++17 > core language support would automatically fall out of this (there are some > stragglers for full stdlib support, especially in clang; but that is usually > not an issue). > > Best > H. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/AGUI6B6W55TUXL6SA7KQQGPYYSRJNCFH/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O6GC3IWERSJCOYUUJQYEFO5NKR5DS6UI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Using the Python C API in C++
Hi, If you got issues with using the Python C API in C++, please speak up! I'm looking for feedback :-) Extending Python by writing C++ code is now easy with the pybind11 project: https://pybind11.readthedocs.io/ It seems like over the last years, building C++ extensions with the Python C API started to emit more C++ compiler warnings. One explanation may be that converting macros to static inline functions (PEP 670) introduce new warnings, even if the old and the new code is exactly the same. I just discover this issue recently. C and C++ compilers treat static inline functions differently. Macros are treated at legacy code which cannot be fixed, like system headers or old C header files, and so many warnings are made quiet. Static inline functions (defined in header files) are treated as regular code and compilers are more eager to emit warnings. I just modified the Python C API to use C++ static_cast(expr) and reinterpret_cast(expr) if it's used with C++. In C, the regular (type)expr cast (called "old-style cast" by C++ compilers ;-)) is still used as before. I'm also working on adding an unit test to make suite that using the Python C API works with a C++ compiler and doesn't emit compiler warnings: * https://github.com/python/cpython/issues/91321 * https://github.com/python/cpython/pull/32175 In terms of C++ version, it was proposed to target C++11. In the pythoncapi-compat project, I got warnings when the NULL constant is used in static inline functions. I modified the pythoncapi_compat.h header file to use nullptr if used with C++ to fix these compiler warnings. So far, I'm unable to reproduce the issue with , and so I didn't try to address this issue in Python. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/73UTO4MTKVJR7XWXTHLOV3V2L2QKP2TK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
On Wed, Apr 27, 2022 at 5:56 PM Antoine Pitrou wrote: > An environment variable is an easy to influence a program or system > whose inner workings you don't control (for example a system that > spawns child Python processes). And it sounds like a good idea to allow > that given that it improves security? Ok, you changed my mind and I added PYTHONDONTADDPATH0=1 env var. Example: $ ./python -c 'import sys, pprint; pprint.pprint(sys.path)' ['', '/usr/local/lib/python311.zip', '/home/vstinner/python/main/Lib', '/home/vstinner/python/main/build/lib.linux-x86_64-3.11-pydebug', '/home/vstinner/.local/lib/python3.11/site-packages'] $ PYTHONDONTADDPATH0=1 ./python -c 'import sys, pprint; pprint.pprint(sys.path)' ['/usr/local/lib/python311.zip', '/home/vstinner/python/main/Lib', '/home/vstinner/python/main/build/lib.linux-x86_64-3.11-pydebug', '/home/vstinner/.local/lib/python3.11/site-packages'] Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JQHOHQ5VSY3AZBQQXO655YE2YB4AJQV2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
The use case for -P still uses environment variables like PYTHONWARNINGS or PYTHONUTF8. That's why -I (isolated) cannot be used. If there is an use case for a ._pth file importing the site module, maybe a different option can be added, no? Adding -P doesn't prevent that. But it seems like use cases are different enough to justify two options, no? Victor On Tue, Apr 26, 2022 at 11:52 PM Victor Stinner wrote: > The only purpose of proposed -P option is to "not add sys.path[0]". > There are use cases which only need that. > > Victor > > On Tue, Apr 26, 2022 at 8:37 PM Steve Dower wrote: > > > > On 4/26/2022 10:46 AM, Victor Stinner wrote: > > > I propose adding a -P option to Python command line interface to "not > > > add sys.path[0]": > > > https://github.com/python/cpython/pull/31542 > > > > > > See the documentation in the PR for the exact behavior of this option. > > > I prefer to add an environment variable, only pass the option > > > explicitly on the command line. > > > > Another viable option might be to add an option to imply "import site", > > which would work together with -I to: > > * ignore environment variables (-E/-I) > > * omit implicit CWD imports (-I) > > * still process .pth files (-?) > > * still include site-packages and user site-packages in sys.path (-?) > > > > It seems likely that the proposed -P would almost always be used with > > -E, since if you can't control CWD then you presumably can't control > > environment variables either. > > > > The existing ._pth functionality starts by implying -I, and allows > > "import site" in the file to explicitly include site. A command-line > > option matching this behaviour would be consistent. There's also already > > configuration in our structures for import site, so there'd be no need > > to add new fields to public APIs for the option. > > > > The biggest issue I see is that the obvious command line options for > > "import site" are already used to imply "do not import site". But then, > > -P isn't obvious either. Maybe an -X option would suffice? > > > > Cheers, > > Steve > > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RHRRPR2QM6VKN36YGKWNGHE4UQWJXEZC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
Since I didn't get any serious review on my pull request since
February, I created this thread on python-dev to get more people
looking into this issue.
On Wed, Apr 27, 2022 at 5:30 PM Paul Moore wrote:
>
> On Wed, 27 Apr 2022 at 15:32, Victor Stinner wrote:
> >
> > On Tue, Apr 26, 2022 at 11:46 AM Victor Stinner wrote:
> > > I propose adding a -P option to Python command line interface to "not
> > > add sys.path[0]":
> > > https://github.com/python/cpython/pull/31542
> >
> > My plan is to merge this change at 2022-05-05, the day before the
> > Python 3.11 feature freeze,
>
> Why leave it until the last minute? That just makes it harder to
> revert if someone immediately finds a problem with it.
I wrote my PR in February. If it goes wrong, we will have until
October to revert it. The idea is to merge it before beta1 to have 6
months to play with it and check for corner cases.
> It seems very rushed to propose this and implement it days before 3.11
> freeze. If it's been an issue for 11 years, then (a) why didn't anyone
> propose this solution months ago, and (b) surely it can wait another
> year?
Different solutions were proposed over the last 11 years. See for
example: https://bugs.python.org/issue13475
Sadly, no solution was merged into Python, only discussed.
> Steve Dower mentioned some (IMO)
> reasonable points, and you pretty much dismissed them without any
> discussion. That doesn't seem like the right way to handle this. In
> particular, I think the question of how this flag interacts with all
> the other flags and settings that affect sys.path and how "isolated"
> Python is, is an important thing to consider[^1].
See the init_config.rst documentation of my PR: isolated=1 implies
add_path=0 (no behavior change)
https://github.com/python/cpython/pull/31542/files
Running Python with a ._pth file implies isolated=1 and so add_path=0
(no behavior change).
It seems like Steve's proposal is orthogonal, but I don't think that
it's exclusive. We can add a second option, no?
> [^1]: We've had multiple attempts to get locale and UTF8 handling
> right, and now have a mess of flags, environment variables, and
> options that frankly only the experts can understand. I fear that we
> could end up with the same issue for "Python initialisation flags" if
> we don't take a less rushed approach.
The locale encoding, the Python filesystem encoding and the Python
UTF-8 Mode are way more complicated problems. I spent years to fix
issues about these, so I'm well aware of these issue. By the way, I
also designed PEP 587 PyConfig API and I implemented it.
Here the -P option effect is restricted to a single function:
pymain_run_python(). My pull request can be summarized as:
- else if (!config->isolated) {
+ else if (config->add_path0) {
Do you think that we should pay attention to something in specific?
Right now, I propose to not add an environment variable and -P is
unrelated to -E (ignore env vars).
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BFZIGOIVCBNSHBQS7JHKXWWYTENGUWJO/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
Oh sorry, I mean that I prefer to *not* add an environment variable, but I'm not strongly against it. How would the environment varaible be used? A command line option is not enough? Victor On Wed, Apr 27, 2022 at 4:44 PM Antoine Pitrou wrote: > > On Tue, 26 Apr 2022 11:46:41 +0200 > Victor Stinner wrote: > > I prefer to add an environment variable, only pass the option > > explicitly on the command line. > > I don't really understand this sentence, can you rephrase? > > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/L4KBLOSENGMPVQARKWAUXUQKTIP2OFPL/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HOJM3LHUQZBCRK6RZBHIG6H3WGLKRDFT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
On Tue, Apr 26, 2022 at 11:46 AM Victor Stinner wrote: > I propose adding a -P option to Python command line interface to "not > add sys.path[0]": > https://github.com/python/cpython/pull/31542 My plan is to merge this change at 2022-05-05, the day before the Python 3.11 feature freeze, unless someone has a good reason to not add this option or if you consider that we need more time to think about this issue. This issue is being discussed for 11 years, see for example: * https://bugs.python.org/issue13475 * https://discuss.python.org/t/python-flag-envvar-to-not-put-current-directory-on-sys-path-but-dont-ignore-pythonpath/4235 Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SXREVOEEIQL7M5YLTMFIOLLLHXRCI362/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
On Tue, Apr 26, 2022 at 8:37 PM Steve Dower wrote: > The biggest issue I see is that the obvious command line options for > "import site" are already used to imply "do not import site". But then, > -P isn't obvious either. Maybe an -X option would suffice? I propose the short option "-P" rather than a long option like -X dont_add_path0 to be able to use the option in a Unix shebang. Sadly, Unix shebangs don't support long options (it's only possible to use a single long option per shebang, it's not convenient). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UPXCCOO3LGZLVLABHO4SK3T2QCUPH4ZZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Add -P command line option to not add sys.path[0]
The only purpose of proposed -P option is to "not add sys.path[0]". There are use cases which only need that. Victor On Tue, Apr 26, 2022 at 8:37 PM Steve Dower wrote: > > On 4/26/2022 10:46 AM, Victor Stinner wrote: > > I propose adding a -P option to Python command line interface to "not > > add sys.path[0]": > > https://github.com/python/cpython/pull/31542 > > > > See the documentation in the PR for the exact behavior of this option. > > I prefer to add an environment variable, only pass the option > > explicitly on the command line. > > Another viable option might be to add an option to imply "import site", > which would work together with -I to: > * ignore environment variables (-E/-I) > * omit implicit CWD imports (-I) > * still process .pth files (-?) > * still include site-packages and user site-packages in sys.path (-?) > > It seems likely that the proposed -P would almost always be used with > -E, since if you can't control CWD then you presumably can't control > environment variables either. > > The existing ._pth functionality starts by implying -I, and allows > "import site" in the file to explicitly include site. A command-line > option matching this behaviour would be consistent. There's also already > configuration in our structures for import site, so there'd be no need > to add new fields to public APIs for the option. > > The biggest issue I see is that the obvious command line options for > "import site" are already used to imply "do not import site". But then, > -P isn't obvious either. Maybe an -X option would suffice? > > Cheers, > Steve > -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CU75ZRF2UTWGU6OHKHQTWSXRZH52WVOF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Add -P command line option to not add sys.path[0]
Hi,
There are 4 main ways to run Python:
(1) python -m module [...]
(2) python script.py [...]
(3) python -c code [...]
(4) python [...]
(1) and (2) insert the directory of the module/script at sys.path[0].
(3) and (4) insert an empty string at sys.path[0].
This behavior is convenient and is maybe part of Python usability
success: importing a module in the current directory is as easy as
"import other_module" (load other_module.py). But it's also a threat
to security: an attacker can override a stdlib module by creating a
Python script with the same name than a stdlib module, like os.py or
shutil.py.
People learning Python commonly create a file with the same name than
a stdlib module (ex: random.py) and then are clueless in face of an
ImportError exception.
Changing the default behavior was discussed multiple times. No
consensus was reached, maybe because most users like the current
default behavior and are not affected by corner cases (see below).
I propose adding a -P option to Python command line interface to "not
add sys.path[0]":
https://github.com/python/cpython/pull/31542
See the documentation in the PR for the exact behavior of this option.
I prefer to add an environment variable, only pass the option
explicitly on the command line.
Since Python 3.4, there is already the -I ("isolated mode") option:
https://docs.python.org/dev/using/cmdline.html#cmdoption-I
The -I option has other effects like disabling user site directories,
it option doesn't fit use cases of the -P option.
One annoying issue of the Python default behavior is that running a
script in /usr/bin/ as root can create or override .pyc files in the
/usr directory, even in the /usr/bin/ directory. Example of this
surprising and annoying issue:
https://github.com/benjaminp/six/issues/359#issuecomment-996159668
The -P option can be used in #!/usr/bin/python shebang to avoid this issue.
--
An alternative would be to change the default behavior to not add
sys.path[0], and add an option to opt-in for Python 3.10 behavior.
Here are my notes about it:
https://github.com/vstinner/misc/blob/main/cpython/pep_path0.rst
What do you think?
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/IU5Q2AXAURFVDPRWNU3BDFVKV2QX5NOR/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proposal to deprecate mailcap
On Tue, Apr 26, 2022 at 5:47 AM Brett Cannon wrote: > After talking about this in the SC today, we agreed to deprecate mailcap > under the auspices of PEP 594: > https://github.com/python/peps/commit/701999a91dc5f976c00d5bde1510226ebd9c7822 > . Good. I proposed https://github.com/python/cpython/pull/91951 to implement the deprecation in Python 3.11. Fixing or documenting the shell injection vulnerability CVE-2015-20107 is still being discussed at: https://github.com/python/cpython/issues/68966 Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y4IAWAWYTNKSIAVTXJGV2ZMQTFV5WYTT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Fri, Apr 22, 2022 at 2:52 PM Fabio Zadrozny wrote: > Humm, now I'm a bit worried... the approach the debugger is using gets the > PyFrameObject that's about to be executed and changes the > PyFrameObject.f_code just before the execution so that the new code is > executed instead. You can already modify _PyInterpreterFrame.f_code using the internal C API. > From what you're saying the PyFrameObject isn't really used anymore > (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will > this approach still let the debugger patch the code object in the frame > before it's actually executed? There is no public C API to modify the "f_code" attribute of a PyFrameObject. There is only PyFrame_GetCode() *getter*. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RTBUDAEPTZBTAHEO5LV77MHEH7URJP3J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On Wed, Apr 20, 2022 at 1:44 PM Antoine Pitrou wrote: > > For consumers of the C API (C extensions, Cython, pybind11, etc.), > > once most implementation details will be hidden, the C API will become > > way more stable. > > The *API* is quite stable already if you don't use the private/internal > functions. Perhaps you're thinking about the ABI? In Fedora, we update Python early during Python alpha versions, and sadly it's common that many C extensions are incompatible (need to be modified) at each 3.x release. A single minor incompatible change is enough to require changing a C extension. I believe that once the C API will leak less implementation details, changing Python will impact less C extensions. HPy API looks more stable by design: it's way smaller and only expose the bare minimum. I took notes on (Python and C API) incompatible changes, impacting most Python projects and C extensions, from Python 3.7 to Python 3.11: https://github.com/vstinner/vstinner.github.io/blob/pelican/draft/python-incompatible-changes.rst "C API > Porting to Python 3.11" section is quite long, PyFrameObject and PyThreadState structures changed a lot (PyFrameObject moved to the internal C API): https://docs.python.org/dev/whatsnew/3.11.html#id6 "C API > Porting to Python 3.10": https://docs.python.org/dev/whatsnew/3.10.html#id2 "C API > Porting to Python 3.9": https://docs.python.org/dev/whatsnew/3.9.html#id2 "Porting to Python 3.8 > Changes in C API": https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M6MINEKX7XVQ72IVI3D4XTP2O2J5LD3F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
On Wed, Apr 20, 2022 at 10:03 AM Antoine Pitrou wrote: > If the HPy design is the long term goal, why not just recommend that > people use HPy? And keep the C API for expert users with specific > needs that are not accomodated by HPy. > > To me, it seems that trying to change the C API to be "like HPy" is > creating a lot of work, churn and pain for little gain. If you put HPy aside, "Fixing" the C API has multiple advantages for CPython and its (C API) users. For consumers of the C API (C extensions, Cython, pybind11, etc.), once most implementation details will be hidden, the C API will become way more stable. The "C API > Porting to Python x.y" section of What's New in Python x.y should become way shorter. Or at least, the number of impacted C extensions should be way smaller. Sadly, fixing the C API to hide implementation details requires to adapt (modify) C extensions. Even if usually only a few lines should be changed, and the pythoncapi-compat project now automates most of these changes. For CPython, no longer leaking implementation details allow to change "any implementation detail" without getting a heavy and annoying backfire from grumpy users should be very comfortable. In Python 3.9, 3.10 and 3.11 development cycles, we got backfire multiple times, and each time, it was really unpleasant both for CPython core devs and for C extensions maintainers (both have legit concerns and use cases). While these changes should ease the migration to HPy, it's not my goal. HPy requires to add a "ctx" parameter, it's a different API (there are multiple subtle differences). > (and, yes, perhaps HPy needs to be funded or supported by the PSF if it > doesn't advance fast enough) What can be done in practice for that? If I understood correctly, Oracle is sponsoring the project since they want to use HPy for GraalPython (of their GraalVM). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PB7BG7W76BQXP3KS3C6XT5GOFJS24LZB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
6 months ago, I wrote a different document based on HPy Manifesto: "PEP: Taking the Python C API to the Next Level" https://mail.python.org/archives/list/python-dev@python.org/message/RA7Q4JAUEITJBOUAXFEJKRRM2RR3QSZI/ Victor On Mon, Apr 4, 2022 at 5:20 PM Petr Viktorin wrote: > > On 03. 02. 22 1:40, Guido van Rossum wrote: > [...] > > > > I understand the CPython is stuck supporting the de-facto standard C API > > for a long time. But unless we pick a "north star" (as people call it > > nowadays) of what we want to support in say 5-10 years, the situation > > will never improve. > > > > My point about "getting one chance to get it right in the next decade" > > is that we have to pick that north star, so we can tell users which > > horse to bet on. If the north star we pick is HPy, things will be clear. > > If it is evolving the C API things will also be clear. But I think we > > have to pick one, and stick to it so users (i.e., package > > maintainers/developers) have clarity. > > A few months later, here's a draft of a “north star” document. > Is this close to what you had in mind? > > https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9Mqc/edit# > > > Please comment (here or there) as appropriate :) > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/DUWBMGLEYP6VFFT7OMMA6KJNJKTEY47R/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NMMFZSCOEKQDCGUAVIOWV3NYZ5M6NQMJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Do you ever use ceval.c's LLTRACE feature?
Hi, In 2017, I proposed to document the feature, but nobody was volunteer to write the doc (issue still open): https://bugs.python.org/issue25571 I added a bare minimum mention of this feature in the "Python Debug Build" documentation: https://docs.python.org/dev/using/configure.html#python-debug-build A bug was fixed in 2018, so it has at least one user :-) https://bugs.python.org/issue34113 In 2017, adding an API to trace instructions with sys.settrace() was discussed, but the idea got rejected: * https://bugs.python.org/issue29400 * https://mail.python.org/pipermail/python-dev/2017-September/149632.html IMO it would be more flexible than the current "lltrace" implementation which hardcode writing bytecode into C stdout stream which is not convenient and cannot be easily redirected. I have no idea if it would be possible to provide an *efficient* implementation of such hook in ceval.c, even if it's only implemented in debug build. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M6R6EWL2FS5SZC4A3ZDIPZH2LI6G4TFO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Accepting PEP 670 – Convert macros to functions in the Python C API
PEP 670 – Convert macros to functions in the Python C API: https://peps.python.org/pep-0670/ That's great! Thanks to everyone who was involved in PEP 670: the feedback helped Erlend and me to write a better PEP. It was a long journey, Erlend started the first discussion about macros in March 2021! https://discuss.python.org/t/what-to-do-with-unsafe-macros/7771 Victor On Tue, Apr 12, 2022 at 11:22 AM Petr Viktorin wrote: > > Hello, > The Steering Council is **Accepting** the latest iteration of PEP 670: > Convert macros to functions in the Python C API. Congratulations! > > > - Petr, on behalf of the Steering Council -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YK2MOWYS3IOL3BQ3DBG7G7YF3MH7BBQM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Declarative imports
While it's different than you proposal, some people may like this magic "smart imports" project which lazily imports a module when it's used :-) Project: https://pypi.org/project/smart-imports/ Replace: --- import math from my_project import calc # 100500 other imports def my_code(argument, function=calc): return math.log(function(argument)) --- with: --- import smart_imports smart_imports.all() # no any other imports def my_code(argument, function=calc): return math.log(function(argument)) --- I don't like it :-) I prefer explicit imports at the top! Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KLXXAFVPY6RMGJVIFPJ5UNIXNO2YG472/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Thu, Apr 7, 2022 at 12:02 PM Petr Viktorin wrote:
> - This API stays with the regular public API (Include/cpython/), but to
> use it you'll need to #define Py_USING_UNSTABLE_API (name up for
> bikeshedding).
Since there is already something similar called "Py_LIMITED", I
suggest dropping "USING_" for just: "Py_UNSTABLE_API".
> - The functions will be renamed to drop the leading underscore. The old
> names will be available as aliases (using #define) and may be removed
> whenever the API changes. (Ideally, the underscore should always mark
> API that's fully private with no guarantees at all.)
Should functions entering the "unstable API" be documented and tested?
For example, _PyEval_EvalFrameDefault() and
_PyInterpreterState_SetEvalFrameFunc() have no test nor doc.
> This applies to:
>
> - PyCode_New, PyCode_NewWithPosOnlyArgs
It would be nice to update Cython to define the Py_UNSTABLE_API macro
before the macro is required to get the function, since Cython still
uses PyCode_New().
Should we deprecate types.CodeType constructor in the Python API,
since types.CodeType.replace() exists and seems to be a better API
("more stable")?
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/Q6W6G4DNC4JB4YWRCEY32RWZHIAKJSU5/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] About PEPs being discussed on Discourse
Hi, Would it be possible to announce new PEPs on python-dev please? I don't go often to Discourse, like once a month. I don't get any notification by email. I expected new PEPs to be announced on python-dev, but they are not announced here anymore. Is it possible to get Discourse notifications by email, but only for new PEPs? Using mailing lists, it's easy: just select the mailing list that you would like to subscribe to. So I went to Discourse, I click on "New Topics" and I don't see any new PEP... ... But if I go manually to the PEP category, there are 2 new PEPs proposed (PEP 678, PEP 687). In this category, if I click on the bell: it says "You will be notified if someone mentions your @name or replies to you". I can change this parameter to "You will be notified of new topics in this category but not replies to the topics." I don't recall if I changed this parameter manually, but it seems like the choice to only be notified of new topics is a new (i don't think that it existed 1 year ago). I don't recall that I opted in to not be notified of new PEPs. Now I see that Inada-san submitted the PEP 686 to the SC: https://github.com/python/steering-council/issues/118 I didn't read the discussion about this PEP which interest me. I knew that it exists because I saw related issues and pull requests about this PEP, but I didn't go to the discussion because I don't have the habit of visiting Discourse. I guess that it's my fault of not going to Discourse often enough. It's sometimes hard to keep track of everything happening around Python development. The discussions are scattered between multiple communication channels: * Issues * Pull requests * python-dev * python-committers * (private) Discord * Discourse * (public) IRC #python-dev Sometimes, I already confused by the same topic being discussed in two different Discord rooms :-) It's also common that some people discuss on the issue, and other people have a parallel discussion (about the same topic) on the related pull request. There are also Language Summit (sadly, I cannot attend it this year, for the first time) and CPython core dev sprints. Sometimes, a discussion happens on Twitter, but if it becomes serious, it moves to the above communication channels, so it's fine. I'm not going to https://python.zulipchat.com/ anymore, I guess that it has been replaced with Discord. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H27BIU2E2OGIBA5KC4USDNBN4I3RJ5GR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [python-committers] Re: [IMPORTANT] Preparations for 3.11.0 beta 1
On Thu, Apr 7, 2022 at 5:35 AM Inada Naoki wrote: > I just submitted the PEP 686 to the SC. > https://github.com/python/steering-council/issues/118 > > In this PEP, I am proposing: > > a. Small improvement for UTF-8 mode in Python 3.11 > b. Make UTF-8 mode default in Python 3.13. It's easier to approve or reject a PEP if all changes target the same Python version. I'm not sure that (a) changes require a PEP, they are mostly bugfixes and uncontroversial enhancements. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JFW6LZPOYKXYMJXZ6DWZW2IE7I6IAU32/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [python-committers] Re: [IMPORTANT] Preparations for 3.11.0 beta 1
IMO adding locale.getencoding() to Python 3.11 is not controversial and is useful even if PEP 686 is rejected. This function was discussed for 1 year (bpo-43510, bpo-43552, bpo-43557, bpo-47000) and there is an agreement that there is a need for this function. > Making `open(path, encoding="locale")` use locale encoding in UTF-8 mode > (Python 3.10 used UTF-8) If someone explicitly opts in for the "locale encoding", it sounds surprising that the locale (encoding) is ignored and that UTF-8 is used if the Python UTF-8 Mode is enabled. I'm fine with this change. If you want to always UTF-8... Pass explicitly UTF-8: # no surprise, always decode file content from UTF-8 json_file = open(filename, encoding="utf-8") -- I will not comment PEP 686 here. It's being discussed on Discourse: * https://discuss.python.org/t/14435 * https://discuss.python.org/t/14737 Victor On Thu, Apr 7, 2022 at 5:35 AM Inada Naoki wrote: > > Hi, Pablo. > > I just submitted the PEP 686 to the SC. > https://github.com/python/steering-council/issues/118 > > In this PEP, I am proposing: > > a. Small improvement for UTF-8 mode in Python 3.11 > b. Make UTF-8 mode default in Python 3.13. > > (a) is an important change for (b) so I included it in the PEP. > More precisely, (a) contains two changes: > > * Making `open(path, encoding="locale")` use locale encoding in UTF-8 > mode (Python 3.10 used UTF-8) > * Add `locale.getencoding()` that is same to > `locale.getpreferredencoding(False)` but returns locale encoding even > in UTF-8 mode. > > These changes are important for (b). > But they are not a big change needing PEP. > > What should I do? > > * Do not merge anything until PEP accepted. > * Merge (a) without waiting PEP accepted. > * Merge (a) and remove it from the PEP. > > FWI, I and Victor are implementing `locale.getencoding()` for now. > > https://bugs.python.org/issue47000 > https://github.com/python/cpython/pull/32068 > > Regards, > -- > Inada Naoki > ___ > python-committers mailing list -- python-committ...@python.org > To unsubscribe send an email to python-committers-le...@python.org > https://mail.python.org/mailman3/lists/python-committers.python.org/ > Message archived at > https://mail.python.org/archives/list/python-committ...@python.org/message/7E4QEKZ6HNDDPDL76LP3TBBKLAUQ7AHB/ > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KEW2G2O57CNO66QFZ2I3E3D6ILQ67RII/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
Hi,
Steve, Petr: sorry if you feel that I didn't take your feedback in
account, it's not case. Thanks for your valuable feedback. It seems
like there was some misunderstanding.
On Tue, Apr 5, 2022 at 2:49 AM Gregory P. Smith wrote:
> Thanks for bringing this up on python-dev, Victor. That was good. But the
> point of the discussion should've been to continue working with people based
> on the replies rather than proceeding to merge removals of the APIs after
> people said they used them. (echoing Steve and Petr here...)
>
> We discussed this on the steering council today. These APIs were in a weird
> state and despite past decisions at the time of PEP-523 in 2016 they should
> be treated as public-ish rather than entirely private. Because we published a
> document saying "here they are, use them!" and multiple projects have done so
> to good effect.
>
> For 3.11 we'd like those PRs reverted.
Ok, I created https://github.com/python/cpython/pull/32343 to revert
these two PRs. I will merge it as soon as the Python 3.11 release
manager (Pablo) unblocks the main branch (he is currently preparing
the alpha7 release).
On Fri, Apr 1, 2022 at 12:36 PM Steve Dower wrote:
> I don't see any additional discussion on the bug, and the prevailing
> opinion from actual users of this API is that it probably shouldn't
> change,
>From what I understood my change basically only impacts "pydevd"
users. Fabio who works on that project said that he was fine with
that change ("I'm ok with changes"):
https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55ANVKHGG74D62HDBOFLC4EXWJ26/
debugpy, ptvsd, PyDev, PyCharm and VSCode Python use the same code
base: https://github.com/fabioz/PyDev.Debugger
Jason Ansel of TorchDynamo was worried that the API couldn't be used
anymore. I replied that the API remains available. It's just about
adding a few lines of code (that I provided).
For me, all impacted users were made aware and joined the discussion.
It's very rare to be able to reach *all* users impacted by an
incompatible C API change!
If I understood correctly, all questions were replied. For example,
yes, the API remains accessible (with additional code), but no, sadly
the API is not stable (can change at each 3.x.0 major release, but
should not change in 3.x.y bugfix release).
On Fri, Apr 1, 2022 at 12:36 PM Steve Dower wrote:
> and certainly shouldn't become internal without additional
> guarantees about stability.
The discussed API is not stable, Brett repeated that. The internal API
is not stable on purpose. I missed the discussion proposing to design
a stable API for PEP 523.
PEP 523 users are that the API is unstable and seem to be used to
update their code at each major (3.x) Python releases. I'm not
surprised that a debugger which requires a fast low-level access to
Python internals require that.
Or are you talking about not breaking the API in 3.x.y bugfix
releases? Currently, it's an unwritten rule. IMO it's well respected
by all core devs who understand that it's important to not break any
API, including private and internal APIs, in minor bugfix releases. My
changes are unrelated to that.
On Fri, Apr 1, 2022 at 1:24 PM Petr Viktorin wrote:
> Now, the people who'd like a non-breaking solution will now need rush to
> develop and merge one until the next release, or the API breaks for
> users and it'll be too late to do anything about it.
We are talking about two projects (pydevd and TorchDynamo) and there
are 6 months until Python 3.11 final release (PEP 664). We are still
at the alpha phase, no? I don't see why a change only affecting two
projects is a big deal, knowing that these two projects have been made
aware, and I offered a patch that they can apply right now. They don't
need to wait for next October to apply my patch.
Anyway, I will revert my changes (once the main branch is unblocked)
to apply the SC's decision.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/TUKA6R7VFIKAWBA2XA7QRNXKGKXDH3WG/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan wrote: > The changes you've made have been excellent, and the existing 3 categories > (stable public ABI, stable public API, unstable internal API) cover the vast > majority of cases. > > The final case that isn't quite covered yet is to offer a "semi-stable" API > category for use cases that are intrinsically coupled to implementation > details that may change between feature releases, but should remain stable > within a release series. > > The concrete motivating example for the new category is the extra APIs you > need in order to provide an alternative eval loop implementation. > > The internal API category doesn't properly cover that case, as the APIs there > are free to change even in maintenance releases, and setting Py_BUILD_CORE > exposes a lot more than what an alternative eval loop would need. > > Regular public functions may work in some cases, but aren't necessarily > practical in others (such as exposing the internal frame details for use in > alternative eval loops). > > From an implementation PoV, my own suggestion would be to define a new API > tier with an opt-in macro rather than relying solely on documentation or > naming conventions. > > For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under > "Include/cpython/semi_stable/" (I don't like "unstable" as potential > terminology here, since the internal API is already unstable - we're > splitting the difference between that and the long term stability of the full > public API) For me an API is either stable (remains the same forever) or unstable (change time to time). Public API means: stable, documented, tested. Internal API means: unstable, not documented, not tested. I'm not convinced that it's worth it to create something in the middle. If you want to add doc and tests, it should become a public stable API. For example, IMO PyCode_New() (C API) and types.CodeType constructor (Python API) should be moved to the internal C API, likely with a deprecation period. Cython should not use it but a new stable API. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LFE4OV3NYWR4GBMCISZ3H7JH3SEFMX2E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Slowly bend the C API towards the limited API to get a stable ABI for everyone
IMO it would be better to keep the HPy design as the long term goal: * Refer to Python objects with opaque handles * All structures are opaque (with a few exceptions, like PyType_Spec) It will likely take multiple iterations (Python releases) to reach this goal, and incompatible C API changes may need a PEP (like PEP 674), but IMO it's good to keep this goal in mind. Otherwise, it's not easy to understand the rationale for changes like https://peps.python.org/pep-0674/ "PEP 674 – Disallow using macros as l-values". Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3FYHB74CF6XBADFRLUVFV6NUZKXRSBSY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Make HAMT available to python script
PEP 603 adds a frozenmap type to collections, implemented as HAMT. For a read-only *dict*, I proposed PEP 416 "Add a frozendict builtin type" in 2012. It was rejected. https://peps.python.org/pep-0416/ The outcome of this PEP was the addition of the types.MappingProxy type (previously, it already existed but it was somehow hidden: type(int.__dict__)). Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IQMBELHLSWSEYQI7BEPTJTYP47BWTBUV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Make HAMT available to python script
Hi, In 2019, Yury Selivanov, who added HAMT and contextvars to Python, wrote PEP 603 "Adding a frozenmap type to collections": https://peps.python.org/pep-0603/ Sadly, the PEP was stuck in discussions: * https://discuss.python.org/t/pep-603-adding-a-frozenmap-type-to-collections/2318 * https://discuss.python.org/t/pep-603-frozenmap-vs-my-frozendict/2473 I don't think that differences between dict and frozenmap was a blocker issue. In fact, if I recall correctly, there was no blocker issue. Anyway, Yury is also maintaining the "immutables" 3rd party module which implements frozenmap: https://pypi.org/project/immutables/ Victor On Fri, Apr 1, 2022 at 11:37 AM zhang kai wrote: > > Hi, > > HAMT is a very useful immutable mapping type. Currently CPython use it > internally to implement contextvar. Considering immutable data structure is > very useful I hope we can make it available to python script(maybe via > collections module). > > Immutable data structures are fundamental parts of our project. Currently we > use a full-featured python immutable data library called pyrsistent. > Pyrsistent is very powerful, however the map type in it is implemented in > python script not in C. It becomes slow when the data set is relatively large. > > On the other hand, CPython now already has an immutable mapping type in it. I > think maybe it’s a good idea to make it public? Many projects can benefit > from it I believe. > > Here is a talk given by the author of javascript immutable-js library explain > why immutable data structures are powerful: > https://www.youtube.com/watch?v=I7IdS-PbEgI > > Pyristent: https://github.com/tobgu/pyrsistent > > What do you think? > > Cheers, > Kai > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/2WYPX44WBFS2ZMZFNMDFMRPROB7G34JQ/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PQ42TTASV3ZOBJUTDHARTVTPY5EPIW57/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: code.replace() and Python 3.11 exception table
I created https://bugs.python.org/issue47185 to discuss this issue:
either recompute automatically co_exceptiontable, or at least document
the change.
Victor
On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner wrote:
>
> ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
> private C API to the internal C API")
>
> On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico wrote:
> >
> > On Fri, 1 Apr 2022 at 19:51, Victor Stinner wrote:
> > > In Python, sadly the types.CodeType type also has a public constructor
> > > and many projects break at each Python release because the API
> > > changes. Hopefully, it seems like the new CodeType.replace() method
> > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > > better abstraction and closer to what developers need in practice.
> >
> > It certainly has been for me. When I want to do bytecode hackery, I
> > usually start by creating a function with def/lambda, then construct a
> > modified function using f.__code__.replace(). It's the easiest way to
> > ensure that all the little details are correct.
>
> Python 3.11 added the concept of "exception table"
> (code.co_exceptiontable). You have to build this table, otherwise
> Python can no longer catch exceptions :-)
>
> I don't know how to build this exception table. It seems like
> currently there is no Python function in the stdlib to build this
> table.
>
> Example:
> ---
> def f():
> try:
> print("raise")
> raise ValueError
> except ValueError:
> print("except")
> else:
> print("else")
> print("exit func")
>
> def g(): pass
>
> if 1:
> code = f.__code__
> g.__code__ = g.__code__.replace(
> co_code=code.co_code,
> co_consts=code.co_consts,
> co_names=code.co_names,
> co_flags=code.co_flags,
> co_stacksize=code.co_stacksize)
> else:
> g.__code__ = f.__code__ # this code path works on Python 3.11
>
> g()
> ---
>
> Output with Python 3.10 (ok):
> ---
> raise
> except
> exit func
> ---
>
> Output with Python 3.11 (oops):
> ---
> raise
> Traceback (most recent call last):
> ...
> ValueError
> ---
>
> By the way, this change is not documented at all:
>
> * https://docs.python.org/dev/library/types.html#types.CodeType
> * https://docs.python.org/dev/whatsnew/3.11.html
>
> I understand that these changes come from the "Zero cost exception
> handling" change:
> https://bugs.python.org/issue40222
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] code.replace() and Python 3.11 exception table
("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
private C API to the internal C API")
On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico wrote:
>
> On Fri, 1 Apr 2022 at 19:51, Victor Stinner wrote:
> > In Python, sadly the types.CodeType type also has a public constructor
> > and many projects break at each Python release because the API
> > changes. Hopefully, it seems like the new CodeType.replace() method
> > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > better abstraction and closer to what developers need in practice.
>
> It certainly has been for me. When I want to do bytecode hackery, I
> usually start by creating a function with def/lambda, then construct a
> modified function using f.__code__.replace(). It's the easiest way to
> ensure that all the little details are correct.
Python 3.11 added the concept of "exception table"
(code.co_exceptiontable). You have to build this table, otherwise
Python can no longer catch exceptions :-)
I don't know how to build this exception table. It seems like
currently there is no Python function in the stdlib to build this
table.
Example:
---
def f():
try:
print("raise")
raise ValueError
except ValueError:
print("except")
else:
print("else")
print("exit func")
def g(): pass
if 1:
code = f.__code__
g.__code__ = g.__code__.replace(
co_code=code.co_code,
co_consts=code.co_consts,
co_names=code.co_names,
co_flags=code.co_flags,
co_stacksize=code.co_stacksize)
else:
g.__code__ = f.__code__ # this code path works on Python 3.11
g()
---
Output with Python 3.10 (ok):
---
raise
except
exit func
---
Output with Python 3.11 (oops):
---
raise
Traceback (most recent call last):
...
ValueError
---
By the way, this change is not documented at all:
* https://docs.python.org/dev/library/types.html#types.CodeType
* https://docs.python.org/dev/whatsnew/3.11.html
I understand that these changes come from the "Zero cost exception
handling" change:
https://bugs.python.org/issue40222
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/KWSPCLXDHBAP2U4LBSMLQEOC7LREDMPB/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
Hi, Update on this issue: I merged my 2 PRs. https://bugs.python.org/issue46850 The following APIs have been moved to the internal C API: - _PyFrameEvalFunction type - _PyInterpreterState_GetEvalFrameFunc() - _PyInterpreterState_SetEvalFrameFunc() - _PyEval_EvalFrameDefault() If you use any of these API in your debugger/profiler project, you have do add something like the code below to your project: --- #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include #if PY_VERSION_HEX >= 0x030B00A7 # include // _PyInterpreterState_SetEvalFrameFunc() # include // _PyEval_EvalFrameDefault() #endif --- Contact me if you need help to update your affected projects. IMO PEP 523 doesn't have to be updated since it already says that the APIs are private. Since these APIs were added by PEP 523, I documented these changes in What's New in Python 3.11 > C API > Porting to Python 3.11,even if these APIs are private. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DNJC6U36CDA7S7ATEGAMUPABBSEIYHC4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum wrote: > In the not so distant past I have proposed to introduce a new category, > "Unstable APIs". These are public but are not guaranteed to be backwards > compatible in feature releases (though I feel they should remain so in bugfix > releases). > > I'm not sure whether those should have a leading underscore or not. Perhaps > (like some other languages do and like maybe we've used in a few places) the > name could just include the word "Unstable"? I recall discussions about PyCode_New(). IMO this API should not be public at all. It leaks way too many implementation details: cell variables, optimization for bytecode offset to line and column numbers, exception table, etc. This API changed often and will continue to change. It's not the right abstraction level. We just exposed the function because it was technically possible and it was convenient since Python consumes its own C API. The internal C API was created to draw a line between what API can be consumed outside Python (public) and what API must not be used outside Python (internal) unless you're writing a debugger or other uncommon very specific use case. The main difference is the warranties provided (public) or not (internal) by Python: tests, documentation, backward compatibility. In Python, sadly the types.CodeType type also has a public constructor and many projects break at each Python release because the API changes. Hopefully, it seems like the new CodeType.replace() method added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a better abstraction and closer to what developers need in practice. I'm not convinced that advertising an API as being Unstable (in the documentation?) is going to solve any kind of problem. People love to use private APIs, and they do it simply because it's technically possible :-) At the end of the day, we have to help them updating their code, otherwise we (at least my Red Hat team) cannot update Python. I designed the internal C API to be more annoying to be used (need to define a macro, need more specific #include) in the hope that people will think twice before using it :-) Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin wrote: > Perhaps we need a new "tier" of C API for debuggers -- API that's > guaranteed stable for a major release, and if it's changed it should > always break with compile errors (e.g. the function gets a new > argument), rather than silently change semantics. > The internal API Cython & greenlet need might go it this category too. We should provide public C API functions for Cython and greenlet needs: well tested and documented functions. There is an on-going work in Python 3.11: * https://docs.python.org/dev/c-api/frame.html * https://github.com/faster-cpython/ideas/issues/309 * https://bugs.python.org/issue40421 Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H64KULVN5O4MXPWWYMO2VBJRTLNNWBYX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Wed, Mar 30, 2022 at 9:26 PM Sebastian Berg wrote: > That is fair, although there are documented underscored ones: > https://docs.python.org/3/search.html?q=_Py > > I suppose that means all bets are off _unless_ it is documented or > later adopted as stable API (e.g. `_PyObject_Vectorcall`). > > With that, the only "not obviously OK" use in NumPy that I am aware of > is `_Py_HashDouble` (it seems undocumented). > > Maybe "unless documented" is just a clear enough distinction in > practice. > Although, to some degree, I think it would be clearer if symbols that > have a realistic chance of changing in bug-fix releases had an > additional safe-guard. Since Python 3.7, there is a work-in-progress to (1) better hide internal C API functions and to (2) promote *private* C API functions being used by 3rd party projects as documented and well tested *public* functions. An example of (2) is the addition of float pack/unpack public functions, like PyFloat_Pack8() and PyFloat_Unpack8(): https://docs.python.org/dev/c-api/float.html#pack-and-unpack-functions There were previously known as private _PyFloat_Pack8() and _PyFloat_Unpack8() functions. They are used by a few serialization projects like msgpack. When they were made public, tests were added and the existing comments were converted to documentation and enhanced. I discovered that these private functions were used when I moved them to the internal C API in bpo-46906 (1) and it broke a few projects. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DT2PUGWPOSZOJZLR4VOMPS6QOS3PDRYR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
The internal C API can be used on purpose. But there is no backward compatibility warranty and it can change anytime. In practice, usually it only changes in 3.x.0 releases. For example, these private C API changed in Python 3.9 and Python 3.11 (see my first email in the other PEP 523 thread). To use the internal C API, you have to declare the Py_BUILD_CORE macro and include an internal C API header file. For _PyInterpreterState_SetEvalFrameFunc(), it should be: #ifndef Py_BUILD_CORE_MODULE # define Py_BUILD_CORE_MODULE #endif #include #include // _PyInterpreterState_SetEvalFrameFunc() #include // _PyEval_EvalFrameDefault Victor On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev wrote: > > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this > proposal may break the next major release of PyTorch. > > The related project is TorchDynamo, which can be found here: > https://github.com/facebookresearch/torchdynamo > > We will likely move this into the core of PyTorch closer to release. > > If the changed happens, would PyTorch still be able to use the eval frame > API? Or would it prevent from being used entirely? > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Wed, Mar 30, 2022 at 2:26 AM Steve Dower wrote: > Right now, the API is allowed/expected to change between 3.x releases > (which normally we don't allow without a deprecation period) but it > still has to remain compatible within a single 3.x release. Making it > fully internal *without adding a stability guarantee* means it could > change more frequently, which you wouldn't be able to handle as an > installable package. > > It's *unlikely* that it'll change that often, because there are still > other public interfaces that cannot. But, the plan behind this is to > make more stuff internal so that it can be modified more freely, so we > may see that rate of change increase. Well, my plan is not break these internal C API just for fun. It's only to better "advertise" the separation between the "stable" public C API (backward compatibility warranty) and the "unstable" private/internal C API (can change any time, changes are not documented). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OMGUI5N33BG3EU4OG3IUZXJQF6XU7X2B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
On Tue, Mar 22, 2022 at 7:33 PM Steve Dower wrote: > After a normal deprecation period, yes? There is no backward compatibility warranty and no deprecation process for private APIs. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GTHWKT3QPPMU4VEC6SNO5HLRNNAZI6US/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
Hi, I proposed two PRs to move the private C API (Include/cpython/) of PEP 523 "Adding a frame evaluation API to CPython" to the internal C API (Include/internals/): * https://github.com/python/cpython/pull/32052 * https://github.com/python/cpython/pull/32054 API: * _PyFrameEvalFunction type * _PyInterpreterState_GetEvalFrameFunc() * _PyInterpreterState_SetEvalFrameFunc() * (undocumented) _PyEval_EvalFrameDefault() The private API to get/set the eval function *is* documented at: https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc I added the Get/Set functions so debuggers don't have to access directly to the PyInterpreterState structure which has been moved to the internal C API in Python 3.8. This API causes me multiple issues: * It's a private API and I'm trying to remove the private API from the public C API header files. * The _PyFrameEvalFunction type is not stable: it got a new "tstate" parameter in Python 3.9 and the type of the second parameter changed from PyFrameObject* to _PyInterpreterFrame* in Python 3.11. * These functions use the _PyInterpreterFrame type which is part of the internal C API. While Pyston didn't bring a JIT compiler to Python with PEP 523, debuggers were made faster by using this API. Debuggers like pydevd, debugpy and ptvsd use it. I propose to move theses API to the internal header files (Include/internals/) to clarify that it's not part of the public C API and that there is no backward compatibility warranty. The change is being discussed at: https://bugs.python.org/issue46850 -- PEP 523 API added more private functions for code objects: * _PyEval_RequestCodeExtraIndex() * _PyCode_GetExtra() * _PyCode_SetExtra() The _PyEval_RequestCodeExtraIndex() function seems to be used by the pydevd debugger. The two others seem to be unused in the wild. I'm not sure if these ones should be moved to the internal C API. They can be left unchanged, since they don't use a type only defined by the internal C API. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] SC accepted PEP 594: Removing dead batteries from the standard library
Hi, Oh, the Steering Council accepted PEP 594 "Removing dead batteries from the standard library"! I just saw the announcement on Discourse. Congratulations Christian and Brett! This PEP, first proposed in 2019, wasn't an easy one. https://peps.python.org/pep-0594/ Gregory P. Smith's message on Discourse: """ On behalf of the Python Steering Council, We are accepting PEP-594 32 Removing dead batteries from the standard library. It removes a non-controversial set of very old unmaintained or obsolete libraries from the Python standard library. We expect this PEP to be a one time event, and for future deprecations to be handled differently. One thing we’d like to see happen while implementing it: Document the status of the modules being deprecated and removed and backport those deprecation updates to older CPython branch documentation (at least back to 3.9). That gets the notices in front of more people who may use the docs for their specific Python version. Particular care should also be taken during the pre-release cycles that remove deprecated modules. If it turns out the removal of a module proves to be a problem in practice despite the clear deprecation, deferring the removal of that module should be considered to avoid disruption. Doing a “mass cleanup” of long obsolete modules is a sign that we as a project have been ignoring rather than maintaining parts of the standard library, or not doing so with the diligence being in the standard library implies they deserve. Resolving ongoing discussions around how we define the stdlib for the long term does not block this PEP. It seems worthwhile for us to conduct regular reviews of the contents of the stdlib every few releases so we can avoid accumulating such a large pile of dead batteries, but this is outside the scope of this particular PEP. – Greg for the PSC """ https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-the-standard-library/13508/21 Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HZXXAHW6K65UTNI2BXWBF5G4XNM644YM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Defining tiered platform support
It would be great to have the list of supported platforms per Python version! Maybe supporting new platforms and dropping support for a platform should be document in What's New in Python x.y. GCC does that for example. It also *deprecates* support for some platforms. Example: https://gcc.gnu.org/gcc-9/changes.html -- It's always hard for me to know what is the minimum supported Windows version. PEP 11 refers to Windows support: https://peps.python.org/pep-0011/#microsoft-windows But I don't know how to get this info from the Microsoft documentation. I usually dig into Wikipedia articles to check which Windows version is still supported or not, but I'm confused between "mainstream support" and "extended support". For example, which Python version still support Windows 7? Wikipedia says that Windows 7 mainstream support ended in 2015, and extended support ended in 2020. But Python still has a Windows 7 SP1 buildbot for Python 3.8: https://buildbot.python.org/all/#/builders/60 What is the minimum Windows supported by Python 3.10? Victor On Mon, Mar 7, 2022 at 8:06 PM Christian Heimes wrote: > > On 07/03/2022 18.02, Petr Viktorin wrote: > >> Why the devguide? I view the list of platforms as important for public > >> consumption as for the core dev team to know what to (not) accept PRs > >> for. > > > > So, let's put it in the main docs? > > Yes, I guess the devguide is a weird place to check for this kind of > > info. But a Python enhancement proposal is even weirder. > > > +1 for our main docs (cpython/Doc/) > > Platform support is Python versions specific. Python 3.10 may support > different version than 3.11 or 3.12. It makes sense to keep the support > information with the code. > > Christian > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/LJID7Y7RFSCRUYLJS3E56WBGJU2R44E4/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JJRNJWNRYC4R47YB64D6ZGO4325WG32R/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Improvements to the sys.path initialization documentation
On Fri, Mar 4, 2022 at 1:37 PM Eryk Sun wrote: > I don't understand. The site packages directories, including virtual > environments, are a site extension. I propose to change that. Move all code related to sys.path from the site module to the Modules/getpath.py module. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HXD2JIDHBX7BFIU3FA7R3LWRFEXQMVYW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Improvements to the sys.path initialization documentation
Unrelated to the doc, now that getpath has been rewritten in pure Python, thanks to Steve Dower!, it would be nice to move the last bits of the sys.path initialization from the site module to the getpath module. It's unpleasant to have a different sys.path depending if the site module is loaded or not. site handles venv, computes absolute paths and adds the user directory. Loading customization modules (sitecustomize and usercustomize) should stay in the site module. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4GZZ5PRP5YLZUVE34NYQQNUF4KBNMNNV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [python-committers] Re: Python 3.11.0a6 is blocked
By the way, AMD64 Arch Linux Usan 3.x started failing because I enabled more tests on this buildbot yesterday. Previously, "test_faulthandler test_hashlib test_concurrent_futures test_ctypes" were simply skipped on this UBSAN buildbot. I'm working on fixing the 3 failing tests: test_ctypes, test_faulthandler and test_hashlib https://bugs.python.org/issue46913 So it's not a regression, it's just that the buildbot became stricter. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G7JH45WMFY3NNWXH3N4OERT62HVOAMCT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Improvements to the sys.path initialization documentation
Hi, I tried to document inputs and outputs of Modules/getpath.py in the C API Initialization API: https://docs.python.org/dev/c-api/init_config.html#python-path-configuration I don't know if it would be possible to merge most of these information at a single place to avoid oudated documentation. Victor On Wed, Mar 2, 2022 at 5:40 PM Russel Webber wrote: > > Hi All, > > Steve Dower, Eryk Sun and I have been working on improving the documentation > around how sys.path is initialised and therefore how Python modules are found. > > We're moving the details from > https://docs.python.org/3.11/using/windows.html#finding-modules > to a platform-agnostic new section in > https://docs.python.org/3.11/using/cmdline.html > > We're aiming to have one place in the documentation where all the ways to > customize sys.path (PYTHONPATH, ._pth, site, etc) are at least mentioned. > > We'd appreciate further community feedback on the changes before they are > merged. > https://bugs.python.org/issue31582 > https://github.com/python/cpython/pull/31082 > > Thank you, > Russel > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/5AZQZH45EUCYN26SVD5DO6O7XTPSTIB5/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q5VGOJPJE67JKCMZO2CZBH6UFM7EMXDY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Should we require IEEE 754 floating-point for CPython?
The consensus is to require IEEE 754 to build CPython, but not require it in the Python language specification. Updates (changed merged in bpo-46656): * Building Python 3.11 now requires a C11 compiler without optional C11 features. I wrote it in What's New in Python 3.11 and the PEP 7. * Building Python 3.11 now requires support for floating point Not-a-Number (NaN): remove the Py_NO_NAN macro. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EP625OXOLAQ3DSGFWICYAGVKAPWDSF2V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
On Thu, Feb 24, 2022 at 11:10 PM Barry wrote: > > "Python 3.11 and newer versions use C11 without optional features. The > > public C API should be compatible with C++." > > https://github.com/python/peps/pull/2309/files > > Should is often read as meaning optional when writing specs. > Can you say “must be compatible with C++”. I plan to attempt to write an actual test for that, rather than a vague sentence in a PEP. For now, "should" is a deliberate choice: I don't know exactly which C++ version should be targeted and if it's really an issue or not. For example, C++20 reserves the "module" keyword, whereas Python uses it in its C API. Example: PyAPI_FUNC(int) PyModule_AddType(PyObject *module, PyTypeObject *type); See: * https://bugs.python.org/issue39355 * https://github.com/pythoncapi/pythoncapi_compat/issues/21 -- I made a change in the datatable project to add Python 3.11 support using the pythoncapi_compat.h header file. Problem: this *C* header file produced new warnings in datatable extension module which with built with a C++ compiler: https://github.com/h2oai/datatable/pull/3231#issuecomment-1032864790 Examples: | src/core/lib/pythoncapi_compat.h:272:52: warning: zero as null pointer constant [-Wzero-as-null-pointer-constant] ||| tstate->c_profilefunc != NULL); |^~~~ |nullptr and | src/core/lib/pythoncapi_compat.h:170:12: warning: use of old-style cast [-Wold-style-cast] | return (PyCodeObject *)_Py_StealRef(PyFrame_GetCode(frame)); |^ I made pythoncapi_compat.h compatible with C++ (fix C++ compiler warnings) by using nullptr and reinterpret_cast(EXPR) cast if the __cplusplus macro is defined, or NULL and ((TYPE)(EXPR)) cast otherwise. datatable also uses #include "Python.h". I don't know there were only C++ compiler warnings on "pythoncapi_compat.h". Maybe because datatable only uses static inline functions from "pythoncapi_compat.h", but it may also emit the same warnings if tomorrow some static inline functions of "Python.h" are used. For now, I prefer to put a reminder in PEP 7 that the "Python.h" C API is consumed by C++ projects. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RGNBM5CSUPBQSTZND4PHEV3WUEKS36TP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
Ok, let me try something simpler: "Python 3.11 and newer versions use C11 without optional features. The public C API should be compatible with C++." https://github.com/python/peps/pull/2309/files Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/C32XCK5WHMLJNV6RGLMN2XNBDWR4DI3V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Version 2 of PEP 670 – Convert macros to functions in the Python C API
On Wed, Feb 23, 2022 at 7:11 PM Petr Viktorin wrote: > I did realize there's one more issue when converting macros or static > inline functions to regular functions. > Regular functions' bodies aren't guarded by limited API #ifdefs, so if > they are part of the limited API it's easy to forget to think about it > when changing them. > If a macro in the limited API is converted to a regular function, then a > test should be added to ensure the old implementation of the macro (i.e. > what's compiled into stable ABI extensions) still works. Does it problem really belongs to PEP 670 "Convert macros to functions in the Python C API", or is it more something for PEP 652 "Maintaining the Stable ABI"? I don't think that Python 3.11 should keep a copy of Python 3.10 macros: it would increase the maintenance burden, each function would have 2 implementations (3.11 function and 3.10 macro). Also, there would be no warranty that the copied 3.10 macros would remain exactly the same than 3.10 code if someone changes them by mistake directly or indirectly (by changing code used by this macro, changing a compiler flag, etc). Maybe such stable ABI test belongs to an external project building a C extension with the Python 3.10 limited C API (or an older version) and then test it on Python 3.11. IMO it's the reliable way to test the stable ABI: a functional test. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HTO2BIVD2SIJGXY3HC7OFG3YW7PXXTT6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Version 2 of PEP 670 – Convert macros to functions in the Python C API
On Wed, Feb 23, 2022 at 7:11 PM Petr Viktorin wrote: > In the PEP, the "Performance and inlining" section seems unnecessary. It > talks about attributes that aren't used in the implementation. Or are > they? How does the section relate to the rest of the PEP? > The "Benchmark comparing macros and static inline functions" section at > the end should be enough to explain the impact. I added this section to the PEP because some core devs believe that C compilers don't inline static inline functions sometimes and that it causes performance regressions. This section is an answer to that: I checked that static inline functions *are* inlined as expected in practice. The "Benchmark comparing macros and static inline functions" section uses regular benchmarks to confirm that. Forcing the compiler to inline or asking the compiler to not inline has also been discussed multiple times when it was proposed to convert some macros to static inline functions. So I prefer to put it in the PEP to avoid people having to dig into old discussions to have scattered information about that. You may want to dig into links from the PEP to see old discussions. See for example https://bugs.python.org/issue35059 for the first discussion on forcing inline or not. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YVW7ULTTC6FNTOLL2BPNQOOV3NMCH4BK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
Hi, I updated my PEP 7 PR to use C99 in the public C API and "a subset of" C11 in Python internals: "Python 3.11 and newer versions use C99 in the public C API and use a subset of C11 in Python internals. The public C API should be compatible with C++. The C11 subset are features supported by GCC 8.5, clang 8.0, and MSVC of Visual Studio 2017." https://github.com/python/peps/pull/2309/files GCC 8.5 is the version chosen by RHEL 8. It should provide C11 features that we care about. I pickled clang 8.0 because it's had been released in 2019 and so should be available on most operating systems. FreeBSD uses clang by default. FreeBSD 13 uses clang 11. And it seems like we still care about support Visual Studio 2017, even if Visual Studio 2019 and 2022 are available. I chose to not require supporting AIX XLC. Inada-san wrote: "xlclang fully supports C89/C99/C11. xlc fully supports C89/C99, and partially supports C11." I guess that in practice, we can test a PR on buildbots when trying "new shiny" C11 feature. Moreover, if a C11 feature is missing, it's usually not too complicated to use a workaround for C99 and older. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5IUV6IEB2ARZ4R6MCVK7FZFKRLH7M5NV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: SC reply to PEP 674 -- Disallow using macros as l-values
On Tue, Feb 22, 2022 at 1:54 PM Petr Viktorin wrote: > First, add notices to any documentation that using the macro as a > l-value is a deprecated, CPython-specific detail. It should be clear > that the only reason this usage is allowed is for backwards > compatibility, and that alternate implementations of the C API are free > to not allow this. In practice, alternate implementations must implement exactly the same C API than CPython, otherwise they become incompatible with projects affected by PEP 674 (41 projects including major projects like Cython and numpy). Let me quote Matti about the PyPy project: "I think it would be more accurate to say that PyPy, as a small project in the huge Python ecosystem, is trying to support any C-API used widely by the community. If there is a common PyPy voice (after all, we are also a project with many opinions), we don't "not want to drop support" nor "want to drop support" for any commonly used C-API interfaces, rather we want to stay out of this argument and promote alternatives such as CFFI, cppyy, and HPy instead." https://mail.python.org/archives/list/python-dev@python.org/message/3HGX42QFURHCU6O6DOKBXLVTFIU6RDBO/ The whole purpose of PEP 674 is to remove this constraint in alternate implementations *and* in CPython. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NQG7HXB3LG6S4CPGBYCH3IZSTRSECUYE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: SC reply to PEP 674 -- Disallow using macros as l-values
Hi, Thanks for looking into my PEP 674! I don't understand well why Py_SIZE() cannot be changed until Py_SET_SIZE() is available on all supported Python versions (so affected projects don't have to add a compatibility code), whereas it's ok to require a compatibility code to keep support for Python 3.8 and older for the approved Py_TYPE() change (Py_SET_TYPE was added to Python 3.9). Many projects affected by the Py_SIZE() change are already affected by the Py_TYPE() change, and so need a compatibility code anyway: 33% (5/15) of affected projects are affected by Py_SIZE() and Py_TYPE() changes (see below). Not changing Py_SIZE() now avoids adding a compatibility code to the 33% (5/15) of affected projects only affected by the Py_SIZE() change. Either changing Py_TYPE() *and* Py_SIZE(), or none, would make more sense to me. Well, I would prefer to change both, since the work is already done. The change is already part of Python 3.11 and all projects known to be afffected are already fixed. And the Py_TYPE() change was already approved. -- The Py_TYPE() change requires a compatibility code to get Py_SET_TYPE() on Python 3.8 and older, use pythoncapi_compat.h or add: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) #endif The Py_SIZE() change requires a similar compatibility code. For example, boost defines Py_SET_TYPE() *and* Py_SET_SIZE(): #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) # define Py_SET_SIZE(obj, size) ((Py_SIZE(obj) = (size)), (void)0) #endif -- Affected projects from PEP 674. Projects affected by Py_SIZE() and Py_TYPE() changes (5): * guppy3: Py_SET_TYPE(), Py_SET_SIZE(), Py_SET_REFCNT(), use pythoncapi_compat.h * bitarray: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * mypy: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * numpy: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code * boost: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code Projects only affected by the Py_SIZE() change (5): * python-snappy: Py_SET_SIZE(), use pythoncapi_compat.h * recordclass: use custom py_refcnt() and py_set_size() macros * Cython: Py_SET_SIZE(), Py_SET_REFCNT(), custom compatibility code * immutables: Py_SET_SIZE(), use pythoncapi_compat.h * zstd: Py_SET_SIZE(), use pythoncapi_compat.h Projects only affected by Py_TYPE() change (5): * datatable: Py_SET_TYPE(), Py_SET_REFCNT(), use pythoncapi_compat.h * mercurial: Py_SET_TYPE(), use pythoncapi_compat.h * pycurl: Py_SET_TYPE(), custom compatibility code * duplicity: Py_SET_TYPE(), test PY_MAJOR_VERSION and PY_MINOR_VERSION, or use Py_TYPE() as l-value * gobject-introspection: Py_SET_TYPE(), custom compatibility code These examples don't count the larger number of affected projects using Cython which only need to re-run Cython to use Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE(). I would like to add that 100% of the top 5000 PyPI projects are already fixed for PEP 674, but 26 projects need a release including a fix (which will likely happend before Python 3.11 final release). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YQOSKYGDGRKBAFYW2AJQIIMYMQHALUOQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Version 2 of PEP 670 – Convert macros to functions in the Python C API
Hi, Since Erlend and me posted PEP 670 on python-dev last October, we took all feedback (python-dev and Steering Council) in account to clarify the intent of the PEP and to reduce its scope (remove *any* risk of backward compatibility). PEP 670: https://python.github.io/peps/pep-0670/ Changes since the first version: * Stricter policy on not changing argument types and return type. * Better explain why pointer arguments require a cast to not emit new compiler warnings. * Macros which can be used as l-values are no longer modified by the PEP (I wrote PEP 674 just for that). * Macros having multiple return types are no longer modified by the PEP. * Limited C API version 3.11 no longer casts pointer arguments (move towards a cleaner API, but only if you opt in for that!). * No longer remove return values of macros "which should not have a return value" (this rule was complicated and in fact, all macros have already been fixed). * Add "Macros converted to functions since Python 3.8" section to show that macros already converted didn't introduce performance issue nor new compiler warnings. * Add "Benchmark comparing macros and static inline functions" section to prove that there is no significant impact on performance. Note: we already submitted PEP 670 to the Steering Council at the end of November, and the SC 2022 came with feedback, so we modified the PEP: https://mail.python.org/archives/list/python-dev@python.org/message/IJ3IBVY3JDPROKX55YNDT6XZTVTTPGOP/ Following the SC decision, we already modified PEP 7 to add: "static inline functions should be preferred over macros in new code." https://www.python.org/dev/peps/pep-0007/#c-dialect Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VM6I3UHVMME6QRSUOYLK6N2OZHP454W6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
Well, maybe it's a bad example. I just wanted to say that converting macros to static inline functions provide more accurate profiler data and debuggers can more easily show static inline functions names when they are inlined and put breakpoints of them. But you're right, it's not a silver bullet ;-) Victor On Mon, Feb 14, 2022 at 11:29 AM Antoine Pitrou wrote: > > On Wed, 9 Feb 2022 17:49:19 +0100 > Victor Stinner wrote: > > On Wed, Feb 9, 2022 at 1:04 PM Petr Viktorin wrote: > > > > Right now, a large number of macros cast their argument to a type. A > > > > few examples: > > > > > > > > #define PyObject_TypeCheck(ob, type) > > > > PyObject_TypeCheck(_PyObject_CAST(ob), type) > > > > #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i]) > > > > #define PyDict_GET_SIZE(mp) (assert(PyDict_Check(mp)),((PyDictObject > > > > *)mp)->ma_used) > > > > > > When I look at the Rationale points, and for the first three of these > > > macros I didn't find any that sound very convincing: > > > - Functions don't have macro pitfalls, but these simple macros don't > > > fall into the pits. > > > - Fully defining the argument types means getting rid of the cast, > > > breaking some code that uses the macro > > > - Debugger support really isn't that useful for these simple macros > > > - There are no new variables > > > > Using a static inline function, profilers like Linux perf can count > > the CPU time spend in static inline functions (on each CPU instruction > > when using annotated assembly code). For example, you can see how much > > time (accumulated time) is spent in Py_INCREF(), to have an idea of > > the cost of reference counting in Python. > > The "time spent in Py_INCREF" doesn't tell you the cost of reference > counting. Modern CPUs execute code out-of-order and rely on many > internal structures (such as branch predictors, reorder buffers...). > The *visible* time elapsed between the instruction pointer entering and > leaving a function doesn't tell you whether Py_INCREF had adverse > effects on the utilization of such internal structures (making > reference counting more costly than it appears to be), or on the > contrary whether the instructions in Py_INCREF were successfully > overlapped with other computations (making reference counting > practically free). > > The only reliable way to evaluate the cost of reference counting is to > compare it against alternatives in realistic scenarios. > > Regards > > Antoine. > > > > > It's not possible when using > > macros. > > > > For debuggers, you're right that Py_INCREF() and PyTuple_GET_ITEM() > > macros are very simple and it's not so hard to guess that the debugger > > is executing their code in the C code or the assembly code. But the > > purpose of PEP 670 is to convert way more complex macros. I wrote a PR > > to convert unicodeobject.h macros, IMO there are one of the worst > > macros in Python C API: > > https://github.com/python/cpython/pull/31221 > > > > I always wanted to convert them, but some core devs were afraid of > > performance regressions. So I wrote a PEP to prove that there is no > > impact on performance. > > > > IMO the new unicodeobject.h code is way more readable. I added two > > "kind" variables which have a well defined scope. In macros, no macro > > is used currently to avoid macro pitfalls: name conflict if there is > > already a "kind" macro where the macro is used. > > > > The conversion to static inline macros also detected a bug with "const > > PyObject*": using PyUnicode_READY() on a const Unicode string is > > wrong, this function modifies the object if it's not ready (WCHAR > > kind). It also detected bugs on "const void *data" used to *write* > > into string characters (PyUnicode_WRITE). > > > > > > > - Macro tricks (parentheses and comma-separated expressions) are needed, > > > but they're present and tested. > > > > The PEP rationale starts with: > > "The use of macros may have unintended adverse effects that are hard > > to avoid, even for experienced C developers. Some issues have been > > known for years, while others have been discovered recently in Python. > > Working around macro pitfalls makes the macro coder harder to read and > > to maintain." > > > > Are you saying that all core devs are well aware of all macro pitfalls > > and always avoid them? I'm well aware of these pitfalls, and I fall > > in
[Python-Dev] Re: Move the pythoncapi_compat project under the GitHub Python or PSF organization?
Results of the poll (which was open for 10 days): * Move pythoncapi_compat: 19 votes (90%) * Don't move pythoncapi_compat: 2 votes (10%) Victor On Fri, Feb 11, 2022 at 12:16 AM Victor Stinner wrote: > > I created a poll on Discourse: > https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-github-python-or-psf-organization/13651 > > It will be closed automatically in 10 days. > > Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B4T3FRH2F4MV7LXOTIUZHR2CLYMJSHHQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Move the pythoncapi_compat project under the GitHub Python or PSF organization?
On Fri, Feb 11, 2022 at 3:14 PM Petr Viktorin wrote: > Sounds reasonable, but... > > The implication of endorsing code like this is that *we cannot change > private API even in patch releases*, which I don't think is documented > anywhere, and might be a bit controversial. > (I'm still planning to document this along with other similar issues, so > if anyone wants to help here, please coordinate with me: > https://discuss.python.org/t/documenting-python-versioning-and-stability-expectations/11090/4). Maybe... we should not do that :-D The PyGC_Head ABI change was the most controversial, but it was already part of the private API and the following bug was closed as "not a bug": https://bugs.python.org/issue39599 This structure now belings to the *internal* C API. > Since you're now inviting the wider community as co-maintainers, could > you document this? Well, anyone is welcomed to contribute! Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WZTVXVBSYS5J3PITADSIYI32KZEJS2MR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
IMO we need to distinguish the public C API which should be as much compatible as possible, target the oldest C standard, and the Python internals can require a more recent C standard. For example, today maybe it's reasonable to requires C99 for Include/ headers (public .h files) and support C11 for Python internals (private .c files). Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LG5WBBU5JK44IO6UCKSQEUVCJ55X6ZBE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Move the pythoncapi_compat project under the GitHub Python or PSF organization?
On Fri, Feb 11, 2022 at 12:06 PM Petr Viktorin wrote:
> Or will this send a message that core devs should co-maintain the project?
> I personally wouldn't want to maintain it, but it it looks like there
> are at least 3 maintainers who do.
I think that Neal provided a better answer than me :-) It would be
great than once an incompatible C API change is introduced in Python,
pythoncapi_compat is updated. Updating pythoncapi_compat (with unit
tests) ensures that it *is* possible to write a single code base
compatible with old and new Python versions. Sometimes the change can
be modified just a little it to make that transition smoother.
In the past, that's exactly what I did. When I wrote
PyThreadState_EnterTracing(), I started to create a draft PR for
pythoncapi_compat. I merged the pythoncapi_compat PR once the API
landed in Python. I did that for other new C API functions.
> Note that PyFrame_* and PyThreadState_* replace API that was previously
> private (specifically, documented as subject to change at any time).
> I think pythoncapi_compat is a good tool to run if you used code like that!
My goal for pythoncapi_compat is to only support new *public*
functions. The *implementation* can use the private API. So C
extensions can use the new clean API and remains compatible with old
Python versions using private structures and private functions.
For example, the PyThreadState_EnterTracing() implementation uses
PyThreadState.cframe.use_tracing (Python 3.10) or
PyThreadState.use_tracing (Python 3.9 and older), and also
PyThreadState.cframe.tracing, whereas the PyThreadState structure is
considered as "private".
// bpo-43760 added PyThreadState_EnterTracing() to Python 3.11.0a2
#if PY_VERSION_HEX < 0x030B00A2 && !defined(PYPY_VERSION)
PYCAPI_COMPAT_STATIC_INLINE(void)
PyThreadState_EnterTracing(PyThreadState *tstate)
{
tstate->tracing++;
#if PY_VERSION_HEX >= 0x030A00A1
tstate->cframe->use_tracing = 0;
#else
tstate->use_tracing = 0;
#endif
}
#endif
> Note that the upgrade_pythoncapi script uses regular expressions to turn
> e.g. all occurences of `something->interp` to
> `PyThreadState_GetInterpreter(something)`.
> It's fairly simple, but works in practice -- you aren't likely to have
> another struct with a `interp` member. IOW, you do need to do a thorough
> review after running it, as with a PR from a human contributor. But I
> don't think anyone a expects a *fully* automated solution, do they?
Right, currently the script uses regex because it's built in Python:
no third party dependency needed. So far, I didn't have to make any
further changes after the script updated a C extension.
In general, only a few lines number of lines are modified by the
script. Less than 50: 5 to 10 lines on most C extensions.
> The question of maintenance brings op questions about the scope and
> backwards compatibility of pythoncapi_compat itself. Currently there's
> partial support for Python 2.7, and full for 3.5+. Will the fixes stay
> in and accumulate indefinitely?
Supporting 2.7 and 3.5 is cheap: the code is already written. I plan
to continue supporting them until it breaks. Once it will no longer be
possible to test these versions, maybe the code can stay, untested,
and bugs would only be fixed if someone proposes a fix.
It just added Python 3.4 support since it is still possible to get a
Python 3.4 in GitHub Actions (Ubuntu 18.04) :-)
Python 2.7 support is mostly needed by Mercurial which is already
compatible with Python 3. Mercurial plans to drop Python 2 support
soon: in Mercurial 6.2, or maybe even Mercurial 6.1 (the latest
release is 6.0):
https://www.mercurial-scm.org/wiki/Python3
It seems like on Python 3, users target Python 3.6 as the minimum
supported version. Some projects already dropped Python 3.6 support,
move on to Python 3.7.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/XDZUP2ECGFZP5KSJ7ZP5MEJPX7W6UKOK/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Move the pythoncapi_compat project under the GitHub Python or PSF organization?
I created a poll on Discourse: https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-github-python-or-psf-organization/13651 It will be closed automatically in 10 days. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KJ32YNASRWXX7SEY7T6HVMSIGV2DQJC7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Move the pythoncapi_compat project under the GitHub Python or PSF organization?
Hi, Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable? "The pythoncapi_compat project can be used to write a C extension supporting a wide range of Python versions with a single code base. It is made of the pythoncapi_compat.h header file and the upgrade_pythoncapi.py script." Documentation: https://pythoncapi-compat.readthedocs.io/en/latest/ GitHub: https://github.com/pythoncapi/pythoncapi_compat In the past, I managed to move my personal pyperf project under the PSF organization. Now other core developers are contributing and using it. It's better than having it as a personal project. pythoncapi_compat respects the PSF requirements to move a project in the GitHub PSF organization: contributors are required to respect the PSF Code of Conduct and the project has 3 maintainers (Dong-hee Na, Erlend E. AAsland and me). --- Some context. Incompatible C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions. For example, What's New in Python 3.10 suggests adding the following code to your project to get the Py_SET_REFCNT() function on Python 3.9 and older: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0) #endif Python 3.11 requires even more compatibility functions (7) and the implementation of these functions take more lines of code: * Py_SET_TYPE() * Py_SET_SIZE() * PyFrame_GetCode() * PyFrame_GetBack() * PyThreadState_GetFrame() * PyThreadState_EnterTracing() * PyThreadState_LeaveTracing() I added a reference to the pythoncapi_compat project to What's New in Python 3.11: "Or use the pythoncapi_compat project to get these functions on old Python functions." https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes The project provides an upgrade_pythoncapi.py script to automatically add Python 3.10 and 3.11 support without losing support with Python 2.7. Just run the script, maybe copy pythoncapi_compat.h, and you're done! :-) It also provides a "pythoncapi_compat.h" header file which contains all the required compatibility code. Just copy the header file and use >#include "pythoncapi_compat.h"< in your project. You don't have to maintain this compatibilty code yourself anymore (pythoncapi_compat.h is made of 400 lines of C code). pythoncapi_compat supports Python (2.7 to 3.11), PyPy (2.7, 3.6, 3.7) and C++. It is distributed under the MIT license. My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future. --- The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy: https://pythoncapi-compat.readthedocs.io/en/latest/users.html I already made a similar request in June 2021 (move the project under the PSF organization): https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/ What changed since June 2021? Python 3.11 now requires way more compatibility code than Python 3.10 does: 7 functions instead of 1. On Discord, I was asked to ask the Steering Council. I asked the SC who asked me to ask on python-dev, so here I am :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XI23GU54UC2Y6UBMPY6US4IK4PSUXXPI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
On Thu, Feb 10, 2022 at 10:28 AM Petr Viktorin wrote: > Ah, I've also wanted anonymous unions in the past! > There's a little problem in that they're not valid in C++, so we can't > have them in public headers. > > We'll need to mention C++ if we update the standard. IMO we only have to care about C++ in Include/ header files: make sure that "Python.h" is usable in C++. Outside this directory, we don't have to care about C++ compatibility. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AYRNS2PGB6MOUPEY5IJXFMGEVW4T5L6E/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
On Thu, Feb 10, 2022 at 10:58 AM Petr Viktorin wrote:
>
> On 09. 02. 22 21:41, Gregory P. Smith wrote:
> >
> > On Wed, Feb 9, 2022 at 8:54 AM Victor Stinner > Perhaps use a hybrid approach
> > when feasible similar to:
> >#define PyUnicode_CHECK_INTERNED(op)
> > _PyUnicode_CHECK_INTERNED((PyASCIIObject *)(op))
> >
> > That should make it safe. And I believe you do mention this technique
> > as something to be used in the PEP.
It is exactly what the PEP proposes in the Specification section to
avoid adding new compiler warnings:
https://www.python.org/dev/peps/pep-0670/#cast-to-pyobject
Macro which is only there to add the cast, to avoid adding a compiler warning:
#define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob))
Static inline function:
static inline PyTypeObject* _Py_TYPE(const PyObject *ob) { ... }
Use the existing macro to do the cast:
#define _PyObject_CAST_CONST(op) ((const PyObject*)(op))
> That technique is mentioned in the PEP, but it looks like we just found
> a better way to do it than what the PEP suggests: the macro and the
> function can have the same name
Well, when I started to convert macros to static inline functions, I
chose to have a different name to help *me* identifying what is what:
"Pyxxx" is the macro and "_Pyxxx" is the static inline functions. It
looked convenient at the beginning. It looked like a good idea...
Then I started to see "_Pyxxx" functions in debuggers and profilers.
It's not convenient since in C extensions, the public "Pyxxx" (macro)
name is used, same in old Python versions. Having a different name can
cause practical issues: Lib/test/test_gdb.py and
Tools/gdb/libpython.py have to search for multiple names for the same
function in this case (to support old and new Python versions).
The new private/secret "_Pyxxx" name is unexpected, so I proposed to
abandon this bad habit and use the same name for the macro and for the
static inline function.
> Are there any downsides to that?
There is no downside. Macros are only for the C preprocessor, they are
gone at the ABI level. The macro is just there to add a cast to avoid
adding new compiler warnings (since the old macro already did that).
Moreover, PEP 670 proposes to replace some static inline functions
with regular functions to make them available for extension modules
which cannot use static inline functions, like the vim text editor
(written in C) which only loads symbols from libpython. Having a
consistent name for macros, static inline functions and functions
should help for that.
--
There is one exception: when a function as exposed as static inline
function *and* a regular function. _Py_Dealloc() and Py_NewRef() for
example.
Py_NewRef() is defined as a macro which calls _Py_NewRef() static
inline function (for best performance), and there is a regular
Py_NewRef() regular function (exposed in the ABI): the regular
function and the static inline functions cannot have the same name:
PyAPI_FUNC(PyObject*) Py_NewRef(PyObject *obj);
static inline PyObject* _Py_NewRef(PyObject *obj)
{ Py_INCREF(obj); return obj; }
#define Py_NewRef(obj) _Py_NewRef(_PyObject_CAST(obj))
Names:
* Macro: Py_NewRef
* Regular function: Py_NewRef
* Static inline function: _Py_NewRef (different name)
If _Py_NewRef() is renamed to Py_NewRef(), the compiler fails with:
./Include/object.h:597:25: error: static declaration of 'Py_NewRef'
follows non-static declaration
Another example is _Py_Dealloc() declared as a macro+static inline
function (best performance) *and* a regular function (expose it in the
ABI): the static inline function has a different name.
IMO the most important name is the regular function name since it's
the one which lands in libpython ABI. Static inline functions are
*not* part of the libpython ABI, they are either inlined or copied as
regular functions (depending on what the compiler prefers) in each
extension module using it.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/XOLV4BVJDZH3RJRLV3UKDYKA664IANZO/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
On Wed, Feb 9, 2022 at 5:48 PM Pablo Galindo Salgado wrote: > We consider the representation of co_postions private, so we don't want (for > now) to ad > getters/setters. If you want to get the position of a instruction, you can > use PyCode_Addr2Location The code.co_positions() method is accessible in Python: it's not documented, but its name doesn't say that it's private. Was it done on purpose? Should it renamed to _co_positions() or even be removed? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TWATPIZVTXWS224A3Z4VITNU5PKWSEHV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
Hi, It's already possible to call PyObject_CallMethod(code, "co_positions", NULL) and then use the iterator in C. Is there an use case where performance of reading co_positions is critical? If not, there is no need to add a specialized function. Victor On Wed, Feb 9, 2022 at 5:23 PM Petr Viktorin wrote: > > On 04. 02. 22 15:23, Stefan Behnel wrote: > > Petr Viktorin schrieb am 03.02.22 um 13:47: > >> On 02. 02. 22 11:50, Stefan Behnel wrote: > >>> Maybe we should advertise the two modes more. And make sure that both > >>> work. There are certainly issues with the current state of the > >>> "limited API" implementation, but that just needs work and testing. > >> > >> I wonder if it can it be renamed? "Limited API" has a specific meaning > >> since PEP 384, and using it for the public API is adding to the > >> general confusion in this area :( > > > > I was more referring to it as an *existing* compilation mode of Cython > > that avoids the usage of CPython implementation details. The fact that > > the implementation is incomplete just means that we spill over into > > non-limited API code when no limited API is available for a certain > > feature. That will usually be public API code, unless that is really not > > available either. > > > > One recent example is the new error locations in tracebacks, where PEP > > 657 explicitly lists the new "co_positions" field in code objects as an > > implementation detail of CPython. If we want to implement this in > > Cython, then there is no other way than to copy these implementation > > details pretty verbatimly from CPython and to depend on them. > > > > https://www.python.org/dev/peps/pep-0657/ > > > > In this specific case, we're lucky that this can be considered an > > entirely optional feature that we can separately disable when users > > request "public API" mode (let's call it that). Not sure if that's what > > users want, though. > > Should there be a getter/setter for co_positions? > I'm unfortunately not aware of what Cython needs from code objects, but > it might be good to extend the API here. > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/XK4DMU7I35TKXYJRYQE4RGMLNNBPDTYN/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/75CJVEZPQA7ARQDMW62WYYHCIGMNMDP4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
On Wed, Feb 9, 2022 at 1:04 PM Petr Viktorin wrote: > > Right now, a large number of macros cast their argument to a type. A > > few examples: > > > > #define PyObject_TypeCheck(ob, type) > > PyObject_TypeCheck(_PyObject_CAST(ob), type) > > #define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i]) > > #define PyDict_GET_SIZE(mp) (assert(PyDict_Check(mp)),((PyDictObject > > *)mp)->ma_used) > > When I look at the Rationale points, and for the first three of these > macros I didn't find any that sound very convincing: > - Functions don't have macro pitfalls, but these simple macros don't > fall into the pits. > - Fully defining the argument types means getting rid of the cast, > breaking some code that uses the macro > - Debugger support really isn't that useful for these simple macros > - There are no new variables Using a static inline function, profilers like Linux perf can count the CPU time spend in static inline functions (on each CPU instruction when using annotated assembly code). For example, you can see how much time (accumulated time) is spent in Py_INCREF(), to have an idea of the cost of reference counting in Python. It's not possible when using macros. For debuggers, you're right that Py_INCREF() and PyTuple_GET_ITEM() macros are very simple and it's not so hard to guess that the debugger is executing their code in the C code or the assembly code. But the purpose of PEP 670 is to convert way more complex macros. I wrote a PR to convert unicodeobject.h macros, IMO there are one of the worst macros in Python C API: https://github.com/python/cpython/pull/31221 I always wanted to convert them, but some core devs were afraid of performance regressions. So I wrote a PEP to prove that there is no impact on performance. IMO the new unicodeobject.h code is way more readable. I added two "kind" variables which have a well defined scope. In macros, no macro is used currently to avoid macro pitfalls: name conflict if there is already a "kind" macro where the macro is used. The conversion to static inline macros also detected a bug with "const PyObject*": using PyUnicode_READY() on a const Unicode string is wrong, this function modifies the object if it's not ready (WCHAR kind). It also detected bugs on "const void *data" used to *write* into string characters (PyUnicode_WRITE). > - Macro tricks (parentheses and comma-separated expressions) are needed, > but they're present and tested. The PEP rationale starts with: "The use of macros may have unintended adverse effects that are hard to avoid, even for experienced C developers. Some issues have been known for years, while others have been discovered recently in Python. Working around macro pitfalls makes the macro coder harder to read and to maintain." Are you saying that all core devs are well aware of all macro pitfalls and always avoid them? I'm well aware of these pitfalls, and I fall into their trap multiple times. The bpo-30459 issue about PyList_SET_ITEM() is a concrete example of old bugs that nobody noticed before. > The "massive change to working code" part is important. Such efforts > tend to have unexpected issues, which have an unfortunate tendency to > surface before release and contribute to release manager burnout. Aren't you exaggerating a bit? Would you mind to elaborate? Do you have examples of issues caused by converting macros to static inline functions? I'm not talking about incompatible C API changes made on purpose, but about PEP 670 changes. The PEP lists many macros converted to static inline functions and static inline functions converted to regular functions: https://www.python.org/dev/peps/pep-0670/#macros-converted-to-functions-since-python-3-8 Are you aware of release manager burnout caused by these changes? Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LQQDGXEM56J4J7NCSYNJBQPU4JMXHO7B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Steering Council reply to PEP 670 -- Convert macros to functions in the Python C API
Hi Petr,
Thanks for the SC review, it's very helpful! I know that it's a big PEP :-)
On Tue, Feb 8, 2022 at 11:33 AM Petr Viktorin wrote:
> *All other things being equal, static inline functions are better than
> macros.*
> Specifically, inline static functions should be preferred over
> function-like macros in new code. Please add a note about this to PEP 7.
Ok, I created: https://github.com/python/peps/pull/2315
> *When there is no backwards compatibility issue, macros can be changed
> to static inline functions.*
> In effect, the SC accepts the first part of the PEP, except cases where:
> a macro is converted to macro and function ("Cast to PyObject*"),
Right now, a large number of macros cast their argument to a type. A
few examples:
#define PyObject_TypeCheck(ob, type)
PyObject_TypeCheck(_PyObject_CAST(ob), type)
#define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i])
#define PyDict_GET_SIZE(mp) (assert(PyDict_Check(mp)),((PyDictObject
*)mp)->ma_used)
#define PyWeakref_GET_OBJECT(ref) (... ((PyWeakReference
*)(ref))->wr_object ...)
Does it mean that these macros must remain macros?
PEP 670 proposes adding a macro to still cast their arguments to the
requested type so the PEP doesn't add any new compiler warnings. The
PEP proposes to consider removing these implicit casts later (to
reduce the scope and limit annoyance caused by the PEP).
If the macro is converted to a static inline function without such
convenient macro, C extensions which don't pass the expected types
will get spammed by warnings. Well, it's easy to fix, and once it is
fixed, the code remains compatible with old Python versions.
I'm not sure that I understood the SC statement here: does it mean
that these specific macros (which cast their arguments) must remain
macros, or that it's ok to convert them to static inline functions
(even if it can emit new warnings)?
If the SC is ok to add new compiler warnings, I'm fine with it. Most
functions are documented with an exact type, they are not documented
to cast implicitly their arguments to the expected types.
For example, Py_INCREF() expects a PyObject* type in the documentation:
https://docs.python.org/dev/c-api/refcounting.html#c.Py_INCREF
--
Or is the problem that having a macro to wrap a static inline function
requires to change the function name? Well, technically, it's not
needed... I wrote a PR so the macro and the function have the same
name:
https://github.com/python/cpython/pull/31217
For example, the Py_INCREF() macro now calls Py_INCREF(), instead of
calling _Py_INCREF().
Static inline functions are not part of the ABI since they are
inlined, but it's maybe better to avoid the "_" prefix to avoid
confusion in debuggers and profilers (especially when comparing old
and new Python versions).
> and where the return value is removed.
In this case, I propose to leave these macros unchanged in PEP 670 in
this case, as it already excludes macros which can be used as l-values
(macros changed by PEP 674).
I would prefer to have the whole PEP 670 either accepted or rejected.
I'm not comfortable with a half-accepted status, it's misleading for
readers.
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/NHMVI5HSKDQJZG5F457MKFRXRN7GSY2G/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Require a C compiler supporting C99 to build Python 3.11
On Tue, Feb 8, 2022 at 2:02 PM Steve Dower wrote: > All the C99 library is supposedly supported, but there are (big?) gaps > in the compiler support. Some Visual Studio 2019 updates on the Microsoft blog. March 2020: C99 _Pragma https://devblogs.microsoft.com/cppblog/announcing-full-support-for-a-c-c-conformant-preprocessor-in-msvc/ > The _Pragma operator has been one of the long-standing deficiencies of the > preprocessor, and a blocker in being standard conformant in C++ and C99. September 2020: C11 and C17 https://devblogs.microsoft.com/cppblog/c11-and-c17-standard-support-arriving-in-msvc/ Maybe a more practical approach would be to use C99 "except of features not supported by MSVC of Visual Studio 2019"? In practice, we can try to support VS 2017, the version currently recommended by the devguide: https://devguide.python.org/setup/#windows-compiling We already have fallbacks for compilers which don't support C99. We don't have to remove these fallbacks if they are still needed by popular C compilers. For example, I'm fine with keeping basic compatbility support with ICC if it remains reasonable, even if we don't "officially" support ICC (ex: no buildbot). Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L5TIKLYYAHGCSUUWFHXW4ZBHH2TDIHUB/ Code of Conduct: http://python.org/psf/codeofconduct/
