[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again
On Sat, 28 Sep 2019 at 12:56, Victor Stinner wrote: > > Hi, > > I dislike having to do that, but I had to make a last minute change in > my PEP 587 "Python Initialization Configuration" to allow to modify > the structure in the future without breaking the backward > compatibility. I added a "struct_size" field to PyPreConfig and > PyConfig: > > * https://bugs.python.org/issue38304 > * > https://github.com/python/peps/commit/afa38c0bef7738b8fcc3173daee8488e0420833d > > The example: > >PyConfig config; >PyStatus status = PyConfig_InitPythonConfig(&config); >... > > must now be written: > >PyConfig config; >config.struct_size = sizeof(PyConfig); >PyStatus status = PyConfig_InitPythonConfig(&config); >... > > At the beginning, I used a private "_config_version" field which was > initialized statically by a macro. But it was decided to replace > macros with functions. I only noticed today that the conversion to > function broke the API/ABI future compatibility. > > PyConfig_InitPythonConfig() got uninitialized memory and didn't know > the size of the config variable. I don't quite understand the purpose of this change, as there's no stable ABI for applications embedding CPython. As a result, updating to a new X.Y.0 release always requires rebuilding the entire application, not just building and relinking CPython. I could understand a change to require passing in an expected Python version so we can fail more gracefully on a bad link where an application that intended to embed Python 3.8 is incorrectly linked against Python 3.9 (for example), but performing that kind of check would require passing in PY_VERSION_HEX, not the size of the config struct. > With my change, the function now requires the "struct_size" field to > be set, and so it can support internally different versions of the > PyConfig structure. > > Storing the structure size directly in the structure is a common > practice in the Windows API which is a good example of long term ABI > compatibility. This analogy doesn't hold, as Microsoft explicitly support running old binaries on new versions of Windows, and hence need a way to determine which version of the structure is being passed in. We don't support that - all our APIs that accept PyObject/PyTypeObject/etc require that the caller pass in structs of the correct size for the version of Python being used. The PyConfig and PyPreConfig structs are no different from PyObject in that regard: if there's a size mismatch, then the developers of the embedding application have somehow messed up their build process. The stable ABI is a different story, but that's why we try very hard to avoid exposing any structs in the stable ABI. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UHJDLWFHSZ6D7WCHCFAVVBDBK52AS3PL/
[Python-Dev] Re: static variables in CPython - duplicated _Py_IDENTIFIERs?
On Tue., 24 Sep. 2019, 10:29 am Nathaniel Smith, wrote: > On Mon, Sep 23, 2019 at 1:30 PM Vinay Sajip via Python-Dev > wrote: > > > > OK - but that's just one I picked at random. There are others like it - > what would be the process for deciding which ones need to be made private > and moved? Should an issue be raised to track this? > > There are really two issues here: > > - hiding the symbols that *aren't* marked PyAPI_*, consistently across > platforms. > - finding symbols that are currently marked PyAPI_*, but shouldn't be. > > The first one is a pretty straightforward technical improvement. The > second one is a longer-term project that could easily get bogged down > in complex judgement calls. So let's worry about them separately. Even > if there are too many symbols marked PyAPI_*, we can still get started > on hiding all the symbols that we *know* should be hidden. > One of the other things to keep in mind is that a lot of new symbol exports are created by copying an existing one, rather than by going & reading the C API documentation on exporting symbols correctly. Historically, though, it wasn't especially obvious from the header files themselves that "exported for internal linkage" and "exported as part of the public C API" should be written differently, so it was easy to use the wrong style, and patch reviewers wouldn't necessarily notice. With the structural split now in place, the creep of incorrectly published symbols should be contained, and the public header files can start being reviewed for cases like this one. Cheers, Nick. > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7DW34MYR3DZKUXW76DMLB54V22PWMCVX/
[Python-Dev] Re: Compacting the Uncompactable
Short course: a replacement for malloc for use in contexts that can't "move memory" after an address is passed out, but want/need the benefits of compactification anyway. Key idea: if the allocator dedicates each OS page to requests of a specific class, then consider two pages devoted to the same class, where `-` is free space, and X and Y are allocated chunks: page1: X--XX-X---X page2: --Y--Y--YY- Because there's no overlap in the offsets of allocated blocks here, we can copy the 4 Ys into page1 (or Xs into page2). Then tell the OS to change its page tables so that the virtual addresses of page1 amd page2 _both_ map to the physical page page1 referred to. page2's physical page can be returned to the OS then. No _virtual_ address malloc ever handed out needs to change. The rest is making this all safe & "go fast". On Sat, Sep 28, 2019 at 2:06 PM MRAB wrote: > > Here's a video about memory fragmentation and compaction that you might > find interesting: > > "Compacting the Uncompactable" by Bobby Powers > https://www.youtube.com/watch?v=c1UBJbfR-H0 > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/G6OR45TETKIFZDVWAK5ZGLLFTIC422TG/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VMP5RXVQV574SXNJC5SNIJNAL3PYZZC6/
[Python-Dev] Compacting the Uncompactable
Here's a video about memory fragmentation and compaction that you might find interesting: "Compacting the Uncompactable" by Bobby Powers https://www.youtube.com/watch?v=c1UBJbfR-H0 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G6OR45TETKIFZDVWAK5ZGLLFTIC422TG/
[Python-Dev] Re: 3.7.5rc1 release blocked (Re: 3.7.5rc1 cutoff ahead)
2019-09-28 Update: we are still awaiting resolution of some release blockers but I am hopeful we will be unblocked very shortly. I have rescheduled the cutoff for 3.7.5rc1 for this coming Monday (2019-09-30) and, assuming no showstopper problems are identified in the release candidate, 3.7.5 final will follow on 2019-10-14. **Please note** that the next few weeks are **very** important ones for Python releases. Besides 3.7.5, important code cutoffs and release candidate testing periods are coming up for 3.8.0, 3.5.8, and 2.7.17. Please keep an eye out for release blockers, test failures, and any other issues that could affect the quality of releases and please open or update issues on b.p.o, adding **release blocker** priority as necessary to make sure the release managers are aware of them. Thanks! https://www.python.org/dev/peps/pep-0537/ https://www.python.org/dev/peps/pep-0569/ https://www.python.org/dev/peps/pep-0478/ https://www.python.org/dev/peps/pep-0373/ https://bugs.python.org On Sep 17, 2019, at 19:26, Ned Deily wrote: > 2019-09-17 Update: as of the scheduled cutoff time earlier today, we had two > recently identified release blocker issues open. Thanks to Andrew and Yury, > one of them is now resolved with code (bpo-38013). The other (bpo-30458 and > bpo-36274) remains unresolved at this point. Because the issue(s) involve an > apparent regression introduced in a fix for a security issue in 3.7.4, a > regression that has affected at least one third-party project (and has the > potential to affect releases from other branches), I think we should resolve > this now. A number of core devs have been involved in these two issues most > recently Jason. I'm am going to delay tagging the release for at least a > day. Anything you can do to help resolve this one, especially if you have > already been involved with it, would be greatly appreciated. > > https://bugs.python.org/issue30458 > [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of > CVE-2016-5699) > > https://bugs.python.org/issue36274 > http.client cannot send non-ASCII request lines > > > On Sep 9, 2019, at 07:10, Ned Deily wrote: >> https://discuss.python.org/t/3-7-5rc1-cutoff-ahead/2288 >> >> A reminder: it is time for the next quarterly maintenance release of Python >> 3.7. The cutoff for **3.7.5rc1** is scheduled for this coming Monday >> (2019-09-16) by the end of day AOE. Please review open issues and ensure >> that any that you believe need to be addressed in 3.7.5 are either resolved >> or marked as a **release blocker**. Any assistance you can provide in >> helping resolve issues will be greatly appreciated. Following the rc1 >> cutoff, changes merged to the 3.7 branch will be released in 3.7.6 three >> months from now unless you mark the issue as a release blocker prior to >> **3.7.5 final**, planned for release on **2019-09-30** and explain why the >> change should be cherry-picked into the final release. >> >> Thanks to everyone who has been helping to ensure the continued success of >> Python 3.7! Our users truly appreciate it and are showing their confidence >> in us by the rapid adoption of these latest releases. >> >> P.S. A number of core developers are participating in this year's core >> developer sprint taking place this week in London. So it is a good time to >> catch many of us in the same place at the same time (and British Summer >> Time, at that). >> >> https://www.python.org/dev/peps/pep-0537/ -- Ned Deily n...@python.org -- [] ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2HAJ5GZ3K6CR4YREDIR4IZWM4EW4B3LQ/
