[Python-Dev] Re: Using the Python C API in C++
Hi Antoine, I'm in favor of making the C API more strict. The limited C API of Python 3.11 no longer cast arguments to PyObject* ! The caller must now cast to PyObject*: see PEP 670 "convert macros to functions". Converting macros to static inline functions (PEP 670) is already an important change. I prefer to not change too many things at the same time. Maybe in Python 3.12 (or later), we can consider requiring to not pass "const PyObject*" in C++: remove the cast dropping constness of the argument. My long term goal is to only use opaque "PyObject*" "handles" in the C API. But for practical reasons, changing the C API must be done slowly, step by step. To reduce the maintenance burden in third party C and C+ extensions. -- IMO it's a bad idea to use "const PyObject*" in the C API. Maybe today, a function doesn't modify the memory. But maybe tomorrow, the implementation will deeply change and will need to modify the memory. In C, using "const" (for something other than const char*) is just not convenient. Victor On Tue, May 3, 2022 at 9:22 AM Antoine Pitrou wrote: > On Mon, 2 May 2022 15:21:24 +0200 > Victor Stinner wrote: > > > > Slowly, more and more tests can be added. For example, this change fixing > > compiler warnings when passing "const PyObject*" to Python C API > functions > > adds tests to _testcppext.cpp: > > https://github.com/python/cpython/pull/92138 > > Doesn't passing "const PyObject*" leak implementation details, for > example that the reference count does not change? It seems to go > counter the objective of making the C API more abstract and more stable. > > (C++ has the "mutable" keyword for such situat but I don't think C has > it yet) > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/YFD3WOOR6PRORTTKLTSSOSDNHRYAA2N7/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/S3B2KMUL4OM5CCLTYCFCR5HFZBBFWVKV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On Mon, 2 May 2022 15:21:24 +0200 Victor Stinner wrote: > > Slowly, more and more tests can be added. For example, this change fixing > compiler warnings when passing "const PyObject*" to Python C API functions > adds tests to _testcppext.cpp: > https://github.com/python/cpython/pull/92138 Doesn't passing "const PyObject*" leak implementation details, for example that the reference count does not change? It seems to go counter the objective of making the C API more abstract and more stable. (C++ has the "mutable" keyword for such situat but I don't think C has it yet) Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/YFD3WOOR6PRORTTKLTSSOSDNHRYAA2N7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On 5/2/2022 2:21 PM, Victor Stinner wrote: Maybe it should be made optional, but so far I failed to test if distutils has an available C++ compiler. Considering the current status of distutils, you should probably put that check in configure or the Makefile and build the extension from there. Then the test suite can check whether it was built or not. Cheers, Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/46C4ZJZZ5KSPHS4WRICHEOEQZWHXHWWT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Hi, I merged the best basic tests to make sure that using the Python C API in C++ does not emit compiler warnings: * Code: https://github.com/python/cpython/blob/main/Lib/test/_testcppext.cpp * Test: https://github.com/python/cpython/blob/main/Lib/test/test_cppext.py The code is only built to check for compiler warnings. Later, I plan to build this C++ extension to be able to *execute* it at runtime. For now, a C++ compiler is required to run the Python 3.11 test suite. Maybe it should be made optional, but so far I failed to test if distutils has an available C++ compiler. Slowly, more and more tests can be added. For example, this change fixing compiler warnings when passing "const PyObject*" to Python C API functions adds tests to _testcppext.cpp: https://github.com/python/cpython/pull/92138 Victor ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/5I6EJNRWDEHYL773TQHZWHNHG2ZAY3SI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Hi, The C++ version was discussed in the 2nd link that I gave in my first message: https://github.com/python/cpython/issues/91321 Gregory wrote "If we can conditionally test new things based on C++XX version, accumulating modern issue regression tests seems useful. Otherwise 11 at minimum." Another data point is that I mentioned that pybind11 is an important project for Python ecosystem. This project name and its definition refer to C++11: "pybind11: Seamless operability between C++11 and Python". So far, the only C++ code used in Python is the very recent change that I made to use reinterpret_cast<> and static_cast<> in the Python C API (.h header files). The C code base of Python is, well, written only in C. I mentioned nullptr if we get compiler warnings on the NULL constant in static inline functions. nullptr was added by C++11. I also mentioned C++20 "module" keyword and explained that it doesn't affect the Python C API. Having believes and assumptions about C++ compatibility is one thing. My plan is more about actually *testing it* ;-) Victor On Fri, Apr 29, 2022 at 12:36 AM wrote: > > > Not for me to answer, I'm not a proponent of the change. I'm sure if > > you read past discussions here and on Discourse you'll find answers > > from the people who studied the problem carefully. > > The opening mail proposed C++11 without rationale or references. I did search > the archives and discourse before, but nothing stood out, and I don't think an > encyclopedic knowledge of past python-dev discussions is a reasonable > requirement to comment or propose a variation on its merits. > > > I thought you might have something to add to the conversation, but I guess > > not? > > I find this tone quite out-of-place. I made a proposal (based on compiler > support, > version hygiene and compatibility with newer standards), and I'd have been > more > than happy to hear arguments (like Antoine's) or references for the merits of > preferring C++11 (though, again, the point became moot since Victor correctly > pointed out we can test against several versions). > > Still, the insinuation (as it arrives on my end) that I shouldn't participate > seems really unnecessary. > > Best, > H. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/7SSI53EDU2U565O2TYRTU4CPYLVXPO5K/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HV6AV527HVU6REWNFABYFLNV6CHGJ6RN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> Not for me to answer, I'm not a proponent of the change. I'm sure if > you read past discussions here and on Discourse you'll find answers > from the people who studied the problem carefully. The opening mail proposed C++11 without rationale or references. I did search the archives and discourse before, but nothing stood out, and I don't think an encyclopedic knowledge of past python-dev discussions is a reasonable requirement to comment or propose a variation on its merits. > I thought you might have something to add to the conversation, but I guess > not? I find this tone quite out-of-place. I made a proposal (based on compiler support, version hygiene and compatibility with newer standards), and I'd have been more than happy to hear arguments (like Antoine's) or references for the merits of preferring C++11 (though, again, the point became moot since Victor correctly pointed out we can test against several versions). Still, the insinuation (as it arrives on my end) that I shouldn't participate seems really unnecessary. Best, H. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/7SSI53EDU2U565O2TYRTU4CPYLVXPO5K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). Thanks for the datapoint, that's reasonable of course (though I'll note you're using abseil at least through grpc, and abseil is scheduled to remove C++11 support this year: https://abseil.io/blog/20201001-platforms). > We could say that enabling the Python bindings switches the required C++ > version to C++14, but that would bring complication for no actual again > given that you're not likely to benefit from C++14 features in the > header files of a *C* project, are you? I get your point, and I agree. My argument was to ensure compatibility with more recent standard versions, and Victor already suggested that several versions could be tested. Best H. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/X7KTI36ZWSYOURPEG3NFKEFO7TQ5L2WE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On Thu, 28 Apr 2022 17:52:40 +0200 Victor Stinner wrote: > Recently, a issue about C++20 compatibility was reported: > > "The Python library will not compile with a C++2020 compiler because > the code uses the reserved “module” keyword" > https://github.com/python/cpython/issues/83536 > > In fact, after a long study, Python C API is *not* affected by this > issue. Using "module" remains valid in C++20: see the issue for > details. I'm not surprised. The C++ committee takes compatibility extremely seriously... Regards Antoine. > > Victor > > On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou wrote: > > > > On Thu, 28 Apr 2022 22:03:25 +0900 > > "Stephen J. Turnbull" wrote: > > > [email protected] writes: > > > > > > > While I don't know who proposed C++11 or where, I'd therefore like > > > > to propose to move to _at least_ C++14. > > > > > > What benefits does this have for Python development? > > > > Let me second that question as well. > > > > I work on Apache Arrow, where the C++ parts require C++11 (and we can't > > go further than this for now because of R compatibility concerns). We > > could say that enabling the Python bindings switches the required C++ > > version to C++14, but that would bring complication for no actual again > > given that you're not likely to benefit from C++14 features in the > > header files of a *C* project, are you? > > > > Regards > > > > Antoine. > > > > > > ___ > > Python-Dev mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/LCWKZDADMICV3DRNKCLCNDHTXJXTCZ7K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Recently, a issue about C++20 compatibility was reported: "The Python library will not compile with a C++2020 compiler because the code uses the reserved “module” keyword" https://github.com/python/cpython/issues/83536 In fact, after a long study, Python C API is *not* affected by this issue. Using "module" remains valid in C++20: see the issue for details. Victor On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou wrote: > > On Thu, 28 Apr 2022 22:03:25 +0900 > "Stephen J. Turnbull" wrote: > > [email protected] writes: > > > > > While I don't know who proposed C++11 or where, I'd therefore like > > > to propose to move to _at least_ C++14. > > > > What benefits does this have for Python development? > > Let me second that question as well. > > I work on Apache Arrow, where the C++ parts require C++11 (and we can't > go further than this for now because of R compatibility concerns). We > could say that enabling the Python bindings switches the required C++ > version to C++14, but that would bring complication for no actual again > given that you're not likely to benefit from C++14 features in the > header files of a *C* project, are you? > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/AS2CRKHCPLXJ74CTMNURAZ5ANAKUNZ3J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
As a data point, I don't remember that recent versions of CPython brought any particular pain for PyArrow, which is a set of bindings written in Cython around some C++ core library code. Regards Antoine. On Wed, 27 Apr 2022 18:31:13 +0200 Victor Stinner wrote: > Hi, > > If you got issues with using the Python C API in C++, > please speak up! I'm looking for feedback :-) > > Extending Python by writing C++ code is now easy with the pybind11 project: > https://pybind11.readthedocs.io/ > > It seems like over the last years, building C++ extensions with the > Python C API started to emit more C++ compiler warnings. One > explanation may be that converting macros to static inline functions > (PEP 670) introduce new warnings, even if the old and the new code is > exactly the same. I just discover this issue recently. C and C++ > compilers treat static inline functions differently. Macros are > treated at legacy code which cannot be fixed, like system headers or > old C header files, and so many warnings are made quiet. Static inline > functions (defined in header files) are treated as regular code and > compilers are more eager to emit warnings. > > I just modified the Python C API to use C++ static_cast(expr) > and reinterpret_cast(expr) if it's used with C++. In C, the > regular (type)expr cast (called "old-style cast" by C++ compilers ;-)) > is still used as before. > > I'm also working on adding an unit test to make suite that using the > Python C API works with a C++ compiler and doesn't emit compiler > warnings: > > * https://github.com/python/cpython/issues/91321 > * https://github.com/python/cpython/pull/32175 > > In terms of C++ version, it was proposed to target C++11. > > In the pythoncapi-compat project, I got warnings when the NULL > constant is used in static inline functions. I modified the > pythoncapi_compat.h header file to use nullptr if used with C++ to fix > these compiler warnings. So far, I'm unable to reproduce the issue > with , and so I didn't try to address this issue in Python. > > Victor ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JFRJID3ETO3GZ5UHKONDA45WTUS5N7KE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On Thu, 28 Apr 2022 22:03:25 +0900 "Stephen J. Turnbull" wrote: > [email protected] writes: > > > While I don't know who proposed C++11 or where, I'd therefore like > > to propose to move to _at least_ C++14. > > What benefits does this have for Python development? Let me second that question as well. I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you? Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Since we are you talking about tests, we can easily run the tests on multiple C++ versions. But we have to start somewhere, so I propose to start with C++11. More C++ versions can be tested later. Victor On Thu, Apr 28, 2022 at 5:54 AM wrote: > > > In terms of C++ version, it was proposed to target C++11. > > GCC 5 has full C++14 support (one library functionality missing), and so does > VS2015 onwards as well as Clang 3.4, see > https://en.cppreference.com/w/cpp/compiler_support > > I doubt that any older compilers are in use _anywhere_ in reasonable numbers > that this should constrain the development of CPython. > > While I don't know who proposed C++11 or where, I'd therefore like to propose > to move to _at least_ C++14. > > Note that https://github.com/python/peps/pull/2309 already bumped the > required C-standard to C11, and originally defined this as > > The C11 subset are features supported by GCC 8.5, > > clang 8.0, and MSVC of Visual Studio 2017. > > If those versions should be regarded as the lower bounds of compiler support > (are they - or anything lower - tested on the build bots...?), then C++17 > core language support would automatically fall out of this (there are some > stragglers for full stdlib support, especially in clang; but that is usually > not an issue). > > Best > H. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/AGUI6B6W55TUXL6SA7KQQGPYYSRJNCFH/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/O6GC3IWERSJCOYUUJQYEFO5NKR5DS6UI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
[email protected] writes: > > > While I don't know who proposed C++11 or where, I'd therefore like > > > to propose to move to _at least_ C++14. > > > > What benefits does this have for Python development? > > Likewise I can ask what benefits choosing C++11 would have? Not for me to answer, I'm not a proponent of the change. I'm sure if you read past discussions here and on Discourse you'll find answers from the people who studied the problem carefully. I thought you might have something to add to the conversation, but I guess not? Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/2YRVSA4D3DZNPQJBX35P3KW6W7DJGDK4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> > While I don't know who proposed C++11 or where, I'd therefore like > > to propose to move to _at least_ C++14. > > What benefits does this have for Python development? Likewise I can ask what benefits choosing C++11 would have? In general, I think standards and compilers need version hygiene like anything else, just with a larger lag. But even in the standard things get deprecated occasionally, and more importantly: new warnings may get issued, and being able to include CPython into C++ warning-free was one of the points that Victor mentioned specifically. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/KGJN7LN6FWAYFF2IYH5FMWWPEIWH5OMG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
[email protected] writes: > While I don't know who proposed C++11 or where, I'd therefore like > to propose to move to _at least_ C++14. What benefits does this have for Python development? ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HBPHSH5MNHPWQKF6PC5N7VK7X2SWLKI6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> In terms of C++ version, it was proposed to target C++11. GCC 5 has full C++14 support (one library functionality missing), and so does VS2015 onwards as well as Clang 3.4, see https://en.cppreference.com/w/cpp/compiler_support I doubt that any older compilers are in use _anywhere_ in reasonable numbers that this should constrain the development of CPython. While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14. Note that https://github.com/python/peps/pull/2309 already bumped the required C-standard to C11, and originally defined this as > The C11 subset are features supported by GCC 8.5, > clang 8.0, and MSVC of Visual Studio 2017. If those versions should be regarded as the lower bounds of compiler support (are they - or anything lower - tested on the build bots...?), then C++17 core language support would automatically fall out of this (there are some stragglers for full stdlib support, especially in clang; but that is usually not an issue). Best H. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/AGUI6B6W55TUXL6SA7KQQGPYYSRJNCFH/ Code of Conduct: http://python.org/psf/codeofconduct/
