[Python-Dev] Re: Using the Python C API in C++
[email protected] writes: > While I don't know who proposed C++11 or where, I'd therefore like > to propose to move to _at least_ C++14. What benefits does this have for Python development? ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HBPHSH5MNHPWQKF6PC5N7VK7X2SWLKI6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> > While I don't know who proposed C++11 or where, I'd therefore like > > to propose to move to _at least_ C++14. > > What benefits does this have for Python development? Likewise I can ask what benefits choosing C++11 would have? In general, I think standards and compilers need version hygiene like anything else, just with a larger lag. But even in the standard things get deprecated occasionally, and more importantly: new warnings may get issued, and being able to include CPython into C++ warning-free was one of the points that Victor mentioned specifically. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/KGJN7LN6FWAYFF2IYH5FMWWPEIWH5OMG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
[email protected] writes: > > > While I don't know who proposed C++11 or where, I'd therefore like > > > to propose to move to _at least_ C++14. > > > > What benefits does this have for Python development? > > Likewise I can ask what benefits choosing C++11 would have? Not for me to answer, I'm not a proponent of the change. I'm sure if you read past discussions here and on Discourse you'll find answers from the people who studied the problem carefully. I thought you might have something to add to the conversation, but I guess not? Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/2YRVSA4D3DZNPQJBX35P3KW6W7DJGDK4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Since we are you talking about tests, we can easily run the tests on multiple C++ versions. But we have to start somewhere, so I propose to start with C++11. More C++ versions can be tested later. Victor On Thu, Apr 28, 2022 at 5:54 AM wrote: > > > In terms of C++ version, it was proposed to target C++11. > > GCC 5 has full C++14 support (one library functionality missing), and so does > VS2015 onwards as well as Clang 3.4, see > https://en.cppreference.com/w/cpp/compiler_support > > I doubt that any older compilers are in use _anywhere_ in reasonable numbers > that this should constrain the development of CPython. > > While I don't know who proposed C++11 or where, I'd therefore like to propose > to move to _at least_ C++14. > > Note that https://github.com/python/peps/pull/2309 already bumped the > required C-standard to C11, and originally defined this as > > The C11 subset are features supported by GCC 8.5, > > clang 8.0, and MSVC of Visual Studio 2017. > > If those versions should be regarded as the lower bounds of compiler support > (are they - or anything lower - tested on the build bots...?), then C++17 > core language support would automatically fall out of this (there are some > stragglers for full stdlib support, especially in clang; but that is usually > not an issue). > > Best > H. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/AGUI6B6W55TUXL6SA7KQQGPYYSRJNCFH/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/O6GC3IWERSJCOYUUJQYEFO5NKR5DS6UI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On Thu, 28 Apr 2022 22:03:25 +0900 "Stephen J. Turnbull" wrote: > [email protected] writes: > > > While I don't know who proposed C++11 or where, I'd therefore like > > to propose to move to _at least_ C++14. > > What benefits does this have for Python development? Let me second that question as well. I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you? Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
As a data point, I don't remember that recent versions of CPython brought any particular pain for PyArrow, which is a set of bindings written in Cython around some C++ core library code. Regards Antoine. On Wed, 27 Apr 2022 18:31:13 +0200 Victor Stinner wrote: > Hi, > > If you got issues with using the Python C API in C++, > please speak up! I'm looking for feedback :-) > > Extending Python by writing C++ code is now easy with the pybind11 project: > https://pybind11.readthedocs.io/ > > It seems like over the last years, building C++ extensions with the > Python C API started to emit more C++ compiler warnings. One > explanation may be that converting macros to static inline functions > (PEP 670) introduce new warnings, even if the old and the new code is > exactly the same. I just discover this issue recently. C and C++ > compilers treat static inline functions differently. Macros are > treated at legacy code which cannot be fixed, like system headers or > old C header files, and so many warnings are made quiet. Static inline > functions (defined in header files) are treated as regular code and > compilers are more eager to emit warnings. > > I just modified the Python C API to use C++ static_cast(expr) > and reinterpret_cast(expr) if it's used with C++. In C, the > regular (type)expr cast (called "old-style cast" by C++ compilers ;-)) > is still used as before. > > I'm also working on adding an unit test to make suite that using the > Python C API works with a C++ compiler and doesn't emit compiler > warnings: > > * https://github.com/python/cpython/issues/91321 > * https://github.com/python/cpython/pull/32175 > > In terms of C++ version, it was proposed to target C++11. > > In the pythoncapi-compat project, I got warnings when the NULL > constant is used in static inline functions. I modified the > pythoncapi_compat.h header file to use nullptr if used with C++ to fix > these compiler warnings. So far, I'm unable to reproduce the issue > with , and so I didn't try to address this issue in Python. > > Victor ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/JFRJID3ETO3GZ5UHKONDA45WTUS5N7KE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
Recently, a issue about C++20 compatibility was reported: "The Python library will not compile with a C++2020 compiler because the code uses the reserved “module” keyword" https://github.com/python/cpython/issues/83536 In fact, after a long study, Python C API is *not* affected by this issue. Using "module" remains valid in C++20: see the issue for details. Victor On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou wrote: > > On Thu, 28 Apr 2022 22:03:25 +0900 > "Stephen J. Turnbull" wrote: > > [email protected] writes: > > > > > While I don't know who proposed C++11 or where, I'd therefore like > > > to propose to move to _at least_ C++14. > > > > What benefits does this have for Python development? > > Let me second that question as well. > > I work on Apache Arrow, where the C++ parts require C++11 (and we can't > go further than this for now because of R compatibility concerns). We > could say that enabling the Python bindings switches the required C++ > version to C++14, but that would bring complication for no actual again > given that you're not likely to benefit from C++14 features in the > header files of a *C* project, are you? > > Regards > > Antoine. > > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/AS2CRKHCPLXJ74CTMNURAZ5ANAKUNZ3J/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
On Thu, 28 Apr 2022 17:52:40 +0200 Victor Stinner wrote: > Recently, a issue about C++20 compatibility was reported: > > "The Python library will not compile with a C++2020 compiler because > the code uses the reserved “module” keyword" > https://github.com/python/cpython/issues/83536 > > In fact, after a long study, Python C API is *not* affected by this > issue. Using "module" remains valid in C++20: see the issue for > details. I'm not surprised. The C++ committee takes compatibility extremely seriously... Regards Antoine. > > Victor > > On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou wrote: > > > > On Thu, 28 Apr 2022 22:03:25 +0900 > > "Stephen J. Turnbull" wrote: > > > [email protected] writes: > > > > > > > While I don't know who proposed C++11 or where, I'd therefore like > > > > to propose to move to _at least_ C++14. > > > > > > What benefits does this have for Python development? > > > > Let me second that question as well. > > > > I work on Apache Arrow, where the C++ parts require C++11 (and we can't > > go further than this for now because of R compatibility concerns). We > > could say that enabling the Python bindings switches the required C++ > > version to C++14, but that would bring complication for no actual again > > given that you're not likely to benefit from C++14 features in the > > header files of a *C* project, are you? > > > > Regards > > > > Antoine. > > > > > > ___ > > Python-Dev mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > https://mail.python.org/archives/list/[email protected]/message/BIK3SEBQVCX4Y5IX3VDXGSL72P5PWB77/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/LCWKZDADMICV3DRNKCLCNDHTXJXTCZ7K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Decreasing refcount for locals before popping frame
Consider this example code: def test(): a = A() test() Currently, the locals (i.e. `a`) are cleared only after the function has returned: If we attach a finalizer to `a` immediately after the declaration then the frame stack available via `sys._getframe()` inside the finalizer function does not include the frame used to evaluate the function (i.e. with the code object of the `test` function). The nearest frame is that of the top-level module (where we make the call to the function). This is in practical terms no different than: def test(): return A() test() There's no way to distinguish between the two cases even though in the second example, the object is dropped only after the frame (used to evaluate the function) has been cleared. The effect I am trying to achieve is: def test(): a = A() del a Here's a use-case to motivate this need: In Airflow, we're considering introducing some "magic" to help users write: with DAG(...): # some code here That is, without declaring a top-level variable such as `dag`. However, we can't detect the following situation: def create(): with DAG(...) as dag: # some code here create() The DAG is not returned from the function but nevertheless, we can't distinguish between this code and the correct version: def create(): with DAG(...) as dag: # some code here return dag In this case, calling `create` will then "return" the DAG and of course, without a variable assignment, the finalizer will be called – but now we can detect this. I'm thinking that it ought to be possible to clear out `frame->localsplus` before leaving the function frame. I played around with "ceval.c" and only got segfaults. It's complicated machinery :-) Thoughts? ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/D5HCLMN42SIRRUHWPU566R7YYAVLCAEN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
I don't know if there's anything specifically stopping this, but from what I
understand, the precise moment that a finalizer gets called is unspecified, so
relying on any sort of behavior there is undefined and non-portable.
Implementations like PyPy don't always use reference counting, so their garbage
collection might get called some unspecified amount of time later.
I'm not familiar with Airflow, but would you be able to decorate the create()
function to check for good return values? Something like
:import functools
:
:def dag_initializer(func):
:@functools.wraps(func)
:def wrapper():
:with DAG(...) as dag:
:result = func(dag)
:del dag
:if not isinstance(result, DAG):
:raise ValueError(f"{func.__name__} did not return a dag")
:return result
:return wrapper
:
:@dag_initializer
:def create(dag):
:"some code here"
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/EBCLFYZLCTANUYSPZ55GFHG5I7DDTR76/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). Thanks for the datapoint, that's reasonable of course (though I'll note you're using abseil at least through grpc, and abseil is scheduled to remove C++11 support this year: https://abseil.io/blog/20201001-platforms). > We could say that enabling the Python bindings switches the required C++ > version to C++14, but that would bring complication for no actual again > given that you're not likely to benefit from C++14 features in the > header files of a *C* project, are you? I get your point, and I agree. My argument was to ensure compatibility with more recent standard versions, and Victor already suggested that several versions could be tested. Best H. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/X7KTI36ZWSYOURPEG3NFKEFO7TQ5L2WE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
As it has been mentioned there is no guarantee that your variable will even be finalized (or even destroyed) after the frame finishes. For example, if your variable goes into a reference cycle for whatever reason it may not be cleared until a GC run happens (and in some situations it may not even be cleared at any point). The language gives you no guarantees over when or how objects will be finalized or destroyed and any attempt at relying on specific behaviour is deemed to fail because it can change between versions and implementations. On Thu, 28 Apr 2022, 14:14 Malthe, wrote: > Consider this example code: > > def test(): > a = A() > > test() > > Currently, the locals (i.e. `a`) are cleared only after the function > has returned: > > If we attach a finalizer to `a` immediately after the declaration then > the frame stack available via `sys._getframe()` inside the finalizer > function does not include the frame used to evaluate the function > (i.e. with the code object of the `test` function). > > The nearest frame is that of the top-level module (where we make the > call to the function). > > This is in practical terms no different than: > > def test(): > return A() > > test() > > There's no way to distinguish between the two cases even though in the > second example, the object is dropped only after the frame (used to > evaluate the function) has been cleared. > > The effect I am trying to achieve is: > > def test(): > a = A() > del a > > Here's a use-case to motivate this need: > > In Airflow, we're considering introducing some "magic" to help users write: > > with DAG(...): > # some code here > > That is, without declaring a top-level variable such as `dag`. > > However, we can't detect the following situation: > > def create(): > with DAG(...) as dag: > # some code here > > create() > > The DAG is not returned from the function but nevertheless, we can't > distinguish between this code and the correct version: > > def create(): > with DAG(...) as dag: > # some code here > return dag > > In this case, calling `create` will then "return" the DAG and of > course, without a variable assignment, the finalizer will be called – > but now we can detect this. > > I'm thinking that it ought to be possible to clear out > `frame->localsplus` before leaving the function frame. > > I played around with "ceval.c" and only got segfaults. It's > complicated machinery :-) > > Thoughts? > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/D5HCLMN42SIRRUHWPU566R7YYAVLCAEN/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BUT34WUMBSQHKASHDTRSZI5H7GSUAX72/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Using the Python C API in C++
> Not for me to answer, I'm not a proponent of the change. I'm sure if > you read past discussions here and on Discourse you'll find answers > from the people who studied the problem carefully. The opening mail proposed C++11 without rationale or references. I did search the archives and discourse before, but nothing stood out, and I don't think an encyclopedic knowledge of past python-dev discussions is a reasonable requirement to comment or propose a variation on its merits. > I thought you might have something to add to the conversation, but I guess > not? I find this tone quite out-of-place. I made a proposal (based on compiler support, version hygiene and compatibility with newer standards), and I'd have been more than happy to hear arguments (like Antoine's) or references for the merits of preferring C++11 (though, again, the point became moot since Victor correctly pointed out we can test against several versions). Still, the insinuation (as it arrives on my end) that I shouldn't participate seems really unnecessary. Best, H. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/7SSI53EDU2U565O2TYRTU4CPYLVXPO5K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
Dennis Sweeney wrote: > I don't know if there's anything specifically stopping this, but from what I > understand, the precise moment that a finalizer gets called is unspecified, > so relying on any sort of behavior there is undefined and non-portable. > Implementations like PyPy don't always use reference counting, so their > garbage collection might get called some unspecified amount of time later. It's unspecified of course for the language as such, but in the specific case of CPython (which we're targeting), I think the refcounting logic is here to stay and generally speaking, can be relied on. Of course some version may come along to break expectations and I suppose we might cross that bridge when we get to it. > I'm not familiar with Airflow, but would you be able to decorate the create() > function to check for good return values? We could but for the most part, people don't define DAGs inside functions – it happens, but it is not the most simple usage pattern. It's not so much about the function itself, but about being able to determine if a DAG was dropped at the top-level of the module. If the frame clearing behavior was changed so that locals were reclaimed before popping the frame, I think the line number (i.e. `f_lineno`) would have to be that of the function definition, i.e. `def test():` in the examples above. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/FWRP3RPCGXXDQT2IVO7HQBCUQFHGTCRM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
Pablo Galindo Salgado wrote: > As it has been mentioned there is no guarantee that your variable will even > be finalized (or even destroyed) after the frame finishes. For example, if > your variable goes into a reference cycle for whatever reason it may not be > cleared until a GC run happens (and in some situations it may not even be > cleared at any point). I think there is a reasonable guarantee in CPython that it will happen exactly when you leave the frame, assuming there are no cycles or other references to the object. There's always the future, but I don't see a very near future where this will change fundamentally. Relying too much on CPython's behavior is a bad thing, but I think there are cases where it makes sense and can be a pragmatic choice. Certainly lots of programs have successfully relied on `sys._getframe` over the years. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BVO7RMMZ2LJFEG4GRNNTYZU3Q4P3DHV3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
Can you show a run-able example of the successful and unsuccessful usage of `with DAG(): ... `? On Fri, Apr 29, 2022, 6:31 AM Malthe wrote: > Pablo Galindo Salgado wrote: > > As it has been mentioned there is no guarantee that your variable will > even > > be finalized (or even destroyed) after the frame finishes. For example, > if > > your variable goes into a reference cycle for whatever reason it may not > be > > cleared until a GC run happens (and in some situations it may not even be > > cleared at any point). > > I think there is a reasonable guarantee in CPython that it will happen > exactly when you leave the frame, assuming there are no cycles or other > references to the object. There's always the future, but I don't see a very > near future where this will change fundamentally. > > Relying too much on CPython's behavior is a bad thing, but I think there > are cases where it makes sense and can be a pragmatic choice. Certainly > lots of programs have successfully relied on `sys._getframe` over the years. > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/BVO7RMMZ2LJFEG4GRNNTYZU3Q4P3DHV3/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/PK7XVKSI7MSU6IJQIQCWM7BHNO7UT5YW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Decreasing refcount for locals before popping frame
On Fri, 29 Apr 2022 at 06:38, Thomas Grainger wrote:
> Can you show a run-able example of the successful and unsuccessful usage of
> `with DAG(): ... `?
from airflow import DAG
# correct:
dag = DAG("my_dag")
# incorrect:
DAG("my_dag")
The with construct really has nothing to do with it, but it is a
common source of confusion:
# incorrect
with DAG("my_dag"):
...
It is less obvious (to some) in this way that the entire DAG will not
be picked up. You will in fact have to write:
# correct
with DAG("my_dag") as dag:
...
This way, you're capturing the DAG in the top-level scope which is the
requirement.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/HREOTTGPB5JMLGYMIQL4VR2DFI6GBG5J/
Code of Conduct: http://python.org/psf/codeofconduct/
