Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On 2018-11-16, Nathaniel Smith wrote: > [..] it seems like you should investigate (a) whether you can make > Py_LIMITED_API *be* that API, instead of having two different > ifdefs That might be a good idea. One problem is that we might like to make backwards incompatible changes to Py_LIMITED_API. Maybe it doesn't matter if no extensions actually use Py_LIMITED_API. Keeping API and ABI compatibility with the existing Py_LIMITED_API could be difficult. What would be the downside of using a new CPP define? We could deprecate Py_LIMITED_API and the new API could do the job. Also, I think extensions should have to option to turn the ABI compatibility off. For some extensions, they will not want to convert if there is a big performance hit (some macros turn into non-inlined functions, call functions rather than access a non-opaque structure). Maybe there is a reason my toggling idea won't work. If we can use a CPP define to toggle between inline and non-inline functions, I think it should work. Maybe it will get complicated. Providing ABI compatibility like Py_LIMITED_API is a different goal than making the API more friendly to alternative Python VMs. So, maybe it is a mistake to try to tackle both goals at once. However, the goals seem closely related and so it would be a shame to do a bunch of work and not achieve both. Regards, Neil ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Fri, Nov 16, 2018 at 3:12 PM Neil Schemenauer wrote: > Also, the extension module should not take a big performance hit. > So, you can't change all APIs that were macros into non-inlined > functions. People are not going to accept that and rightly so. > However, it could be that we introduce a new ifdef like > Py_LIMITED_API that gives a stable ABI. E.g. when that's enabled, > most everything would turn into non-inline functions. In exchange > for the performance hit, your extension would become ABI compatible > between a range of CPython releases. That would be a nice feature. > Basically a more useful version of Py_LIMITED_API. It seems like a lot of the things being talked about here actually *are* features of Py_LIMITED_API. E.g. it does a lot of work to hide the internal layout of PyTypeObject, and of course the whole selling point is that it's stable across multiple Python versions. If that's the kind of ABI you're looking for, then it seems like you should investigate (a) whether you can make Py_LIMITED_API *be* that API, instead of having two different ifdefs, (b) why no popular extension modules actually use Py_LIMITED_API. I'm guessing it's partly due to limits of the API, but also things like: lack of docs and examples, lack of py2 support, ... -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On 2018-11-16, Brett Cannon wrote: > I think part of the challenge here (and I believe it has been > brought up elsewhere) is no one knows what kind of API is > necessary for some faster VM other than PyPy. I think we have some pretty good ideas as to what are the problematic parts of the current API. Victor's C-API web site has details[1]. We can ask other implementors which parts are hard to support. Here are my thoughts about some desired changes: - We are *not* getting rid of refcounting for extension modules. That would require a whole new API. We might as well start from scratch with Python 4. No one wants that. However, it is likely different VMs use a different GC internally and only use refcounting for objects passed through the C-API. Using refcounted handles is the usual implementation approach. We can make some changes to make that easier. I think making PyObject an opaque pointer would help. - Borrowed references are a problem. However, because they are so commonly used and because the source code changes needed to change to a non-borrowed API is non-trivial, I don't think we should try to change this. Maybe we could just discourage their use? For CPython, using a borrowed reference API is faster. For other Python implementations, it is likely slower and maybe much slower. So, if you are an extension module that wants to work well with other VMs, you should avoid those APIs. - It would be nice to make PyTypeObject an opaque pointer as well. I think that's a lot more difficult than making PyObject opaque. So, I don't think we should attempt it in the near future. Maybe we could make a half-way step and discourage accessing ob_type directly. We would provide functions (probably inline) to do what you would otherwise do by using op->ob_type->. One reason you want to discourage access to ob_type is that internally there is not necessarily one PyTypeObject structure for each Python level type. E.g. the VM might have specialized types for certain sub-domains. This is like the different flavours of strings, depending on the set of characters stored in them. Or, you could have different list types. One type of list if all values are ints, for example. Basically, with CPython op->ob_type is super fast. For other VMs, it could be a lot slower. By accessing ob_type you are saying "give me all possible type information for this object pointer". By using functions to get just what you need, you could be putting less burden on the VM. E.g. "is this object an instance of some type" is faster to compute. - APIs that return pointers to the internals of objects are a problem. E.g. PySequence_Fast_ITEMS(). For CPython, this is really fast because it is just exposing the internal details of the layout that is already in the correct format. For other VMs, that API could be expensive to emulate. E.g. you have a list to store only ints. If someone calls PySequence_Fast_ITEMS(), you have to create real PyObjects for all of the list elements. - Reducing the size of the API seems helpful. E.g. we don't need PyObject_CallObject() *and* PyObject_Call(). Also, do we really need all the type specific APIs, PyList_GetItem() vs PyObject_GetItem()? In some cases maybe we can justify the bigger API due to performance. To add a new API, someone should have a benchmark that shows a real speedup (not just that they imagine it makes a difference). I don't think we should change CPython internals to try to use this new API. E.g. we know that getting ob_type is fast so just leave the code that does that alone. Maybe in the far distant future, if we have successfully got extension modules to switch to using the new API, we could consider changing CPython internals. There would have to be a big benefit though to justify the code churn. E.g. if my tagged pointers experiment shows significant performance gains (it hasn't yet). I like Nathaniel Smith's idea of doing the new API as a separate project, outside the cpython repo. It is possible that in that effort, we would like some minor changes to cpython in order to make the new API more efficient, for example. Those should be pretty limited changes because we are hoping that the new API will work on top of old Python versions, e.g. 3.6. To avoid exposing APIs that should be hidden, re-organizing include files is an idea. However, that doesn't help for old versions of Python. So, I'm thinking that Dino's idea of just duplicating the prototypes would be better. We would like a minimal API and so the number of duplicated prototypes shouldn't be too large. Victor's recent work in changing some macros to inline functions is not really related to the new API project, IMHO. I don't think there is a problem to leave an existing macro as a macro. If we need to introduce new APIs, e.g. to help hide PyTypeObject, those APIs could use inline functions. That
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Fri, 16 Nov 2018 at 17:49, Brett Cannon wrote: > And Just to be clear, I totally support coming up with a totally > stripped-down C API as I have outlined above as that shouldn't be > controversial for any VM that wants to have a C-level API. If a stripped down API like this is intended as "use this and you get compatibility across multiple Python interpreters and multiple Python versions" (essentially a much stronger and more effective version of the stable ABI) then I'm solidly in favour (and such an API has clear trade-offs that allow people to judge whether it's the right choice for them). Having this alongside the existing API, which would still be supported for projects that need low-level access or backward compatibility (or simply don't have the resources to change), but which will remain CPython-specific, seems like a perfectly fine idea. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Brett: > But otherwise I think we are making assumptions here. For me, unless we are > trying to trim the C API down to just what is syntactically supported in > Python and in such a way that it hides all C-level details I feel like we are > guessing at what's best for other VMs, both today and in the future, until > they can tell us that e.g. tuple indexing is actually not a problem > performance-wise. The current API of PyTuple_GET_ITEM() allows to write: PyObject **items = _GET_ITEM(tuple, 0); to access PyTupleObject.ob_item. Not only it's possible, but it's used commonly in the CPython code base. Last week I replaced _GET_ITEM() pattern with a new _PyTuple_ITEMS() macro which is private. To be able to return PyObject**, you have to convert the full tuple into PyObject* objects which is inefficient if your VM uses something different (again, PyPy doesn't use PyObject* at all). More generally, I like to use PyTuple_GET_ITEM() example, just because it's easy to understand this macro. But it's maybe not a good example :-) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Wed, 14 Nov 2018 at 16:09, Gregory P. Smith wrote: > It seems like the discussion so far is: >> >> Victor: "I know people when people hear 'new API' they get scared and >> think we're going to do a Python-3-like breaking transition, but don't >> worry, we're never going to do that." >> Nathaniel: "But then what does the new API add?" >> Greg: "It lets us do a Python-3-like breaking transition!" >> > > That is not what I am proposing but it seems too easy for people to > misunderstand it as such. Sorry. > > Between everything discussed across this thread I believe we have enough > information to suggest that we can avoid an "everyone's afraid of a new 3" > mistake by instead making a shim available with a proposed new API that > works on top of existing Python VM(s) so that if we decide to drop the old > API being public in the future, we could do so *without a breaking > transition*. > I know that has always been my hope, especially if any new API is actually going to be more restrictive instead of broader. > > Given that, I suggest not worrying about defining a new C API within the > CPython project and release itself (yet). > +1 from me. Until we have a PEP outlining the actual proposed API I'm not ready to have it go into 'master'. Helping show the shape of the API by wrapping pre-existing APIs I think that's going to be the way to sell it. > > Without an available benefit, little will use it (and given the function > call overhead we want to isolate some concepts, we know it will perform > worse on today's VMs). > > That "top-5" module using it idea? Maintain forks (hooray for git) of > whatever your definition of "top-5" projects is that use the new API > instead of the CPython API. If you attempt this on things like NumPy, you > may be shocked at the states (plural on purpose) of their extension module > code. That holds true for a lot of popular modules. > > Part of the point of this work is to demonstrate that non-incremental > order of magnitude performance change can be had on a Python VM that only > supports such an API can be done in its own fork of CPython, PyPy, > VictorBikeshedPy, FbIsAfraidToReleaseANewGcVmPy, etc. implementation to > help argue for figuring out a viable not-breaking-the-world transition plan > to do such a C API change thing in CPython itself. > I think part of the challenge here (and I believe it has been brought up elsewhere) is no one knows what kind of API is necessary for some faster VM other than PyPy. To me, the only C API that would could potentially start working toward and promoting **today** is one which is stripped to its bare bones and worst mirrors Python syntax. For instance, I have seen PyTuple_GET_ITEM() brought up a couple of times. But that's not syntax in Python, so I wouldn't feel comfortable including that in a simplified API. You really only need attribute access and object calling to make object indexing work, although for simplicity I can see wanting to provide an indexing API. But otherwise I think we are making assumptions here. For me, unless we are trying to trim the C API down to just what is syntactically supported in Python and in such a way that it hides all C-level details I feel like we are guessing at what's best for other VMs, both today and in the future, until they can tell us that e.g. tuple indexing is actually not a problem performance-wise. And Just to be clear, I totally support coming up with a totally stripped-down C API as I have outlined above as that shouldn't be controversial for any VM that wants to have a C-level API. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Get a running instance of the doc for a PR.
On Sun, 4 Nov 2018 at 14:49, Steven D'Aprano wrote: > On Sun, Nov 04, 2018 at 05:05:07PM +0100, Stephane Wirtel wrote: > > > >If I am making doc patches, shouldn't I be doing that *before* I > > >submit the PR? How else will I know that my changes haven't broken the > > >docs? > > > > You can use the web interface of Github and just add/remove/modify a > > paragraph. > > Does Github show a preview? If not, then my question still stands: how > do I know my changes aren't broken? > > If Github does show a preview, then couldn't the reviewer look at that > too? > GitHub provides a "View" button for files while viewing a diff which will do a basic render for reST files. It won't change all the references like sealso and such, but basic stuff like literals and links will show up appropriately. I think baby steps are probably best here, so getting a zip file as Steve pointed out to start is still an improvement than the status quo and much easier to implement. And any further discussion should probably happen over on core-workflow. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-11-09 - 2018-11-16) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6863 (+29) closed 40154 (+36) total 47017 (+65) Open issues with patches: 2741 Issues opened (46) == #31354: Fixing a bug related to LTO only build https://bugs.python.org/issue31354 reopened by ned.deily #33826: enable discovery of class source code in IPython interactively https://bugs.python.org/issue33826 reopened by xtreak #35177: Add missing dependencies between AST/parser header files https://bugs.python.org/issue35177 reopened by vstinner #35202: Remove unused imports in standard library https://bugs.python.org/issue35202 opened by thatiparthy #35203: Windows Installer Ignores Launcher Installer Options Where The https://bugs.python.org/issue35203 opened by gr-dexterl #35206: [WIP] Add a new experimental _Py_CAPI2 API https://bugs.python.org/issue35206 opened by vstinner #35208: IDLE: Squeezed lines count ignores window width https://bugs.python.org/issue35208 opened by taleinat #35210: Use bytes + memoryview + resize instead of bytesarray + array https://bugs.python.org/issue35210 opened by tzickel #35212: Expressions with format specifiers in f-strings give wrong cod https://bugs.python.org/issue35212 opened by arminius #35213: IDLE: use 'macOS' where appropriate. https://bugs.python.org/issue35213 opened by terry.reedy #35214: Get the test suite passing with clang Memory Sanitizer enabled https://bugs.python.org/issue35214 opened by gregory.p.smith #35216: misleading error message from shutil.copy() https://bugs.python.org/issue35216 opened by cedricvanrompay #35217: REPL history is broken when python is invoked with cmd /c https://bugs.python.org/issue35217 opened by é¶æ¬¸ç¹ #35218: decompressing and then re-compressing zipfiles with Python 3 z https://bugs.python.org/issue35218 opened by keeely #35220: delete "how do I emulate os.kill" section in Windows FAQ https://bugs.python.org/issue35220 opened by deronnax #35221: Enhance venv activate commands readability https://bugs.python.org/issue35221 opened by mdk #35224: PEP 572: Assignment Expressions https://bugs.python.org/issue35224 opened by emilyemorehouse #35225: test_faulthandler fails under ubsan https://bugs.python.org/issue35225 opened by benjamin.peterson #35226: mock.call equality surprisingly broken https://bugs.python.org/issue35226 opened by cjw296 #35227: [RFE] tarfile: support adding file objects without prior known https://bugs.python.org/issue35227 opened by mgorny #35228: Index search in CHM help crashes viewer https://bugs.python.org/issue35228 opened by chrullrich #35231: make install may not call ldconfig on GNU/Linux when using --e https://bugs.python.org/issue35231 opened by pmpp #35232: Add `module`/`qualname` arguments to make_dataclass for pickla https://bugs.python.org/issue35232 opened by Antony.Lee #35234: ssl module falls over with internationalized domain names https://bugs.python.org/issue35234 opened by mcasadevall #35235: Access violation on alloc in Windows x86-64 python, pymalloc_a https://bugs.python.org/issue35235 opened by markind #35236: urllib.request.urlopen throws on some valid FTP files https://bugs.python.org/issue35236 opened by Ian Liu Rodrigues #35238: Alleviate memory reservation of fork_exec in subprocess.Popen https://bugs.python.org/issue35238 opened by oesteban #35240: Travis CI: xvfb-run: error: Xvfb failed to start https://bugs.python.org/issue35240 opened by vstinner #35242: multiprocessing.Queue in an inconsistent state and a traceback https://bugs.python.org/issue35242 opened by szobov #35243: readline timeout too long for async gfx use https://bugs.python.org/issue35243 opened by pmpp #35244: Allow to setup Clang as default compiler for modules build https://bugs.python.org/issue35244 opened by Jaime Torres #35246: asyncio.create_subprocess_exec doesn't accept pathlib.Path lik https://bugs.python.org/issue35246 opened by lilydjwg #35247: test.test_socket.RDSTest.testPeek hangs indefinitely https://bugs.python.org/issue35247 opened by markmcclain #35248: RawArray causes FileNotFoundError at cleanup https://bugs.python.org/issue35248 opened by Mathieu Lamarre #35250: Minor parameter documentation mismatch for turtle https://bugs.python.org/issue35250 opened by Shane Smith #35251: FTPHandler.ftp_open documentation error https://bugs.python.org/issue35251 opened by lys.nikolaou #35252: test_functools dead code after FIXME https://bugs.python.org/issue35252 opened by lys.nikolaou #35253: Linker warning LNK4281 https://bugs.python.org/issue35253 opened by neyuru #35255: delete "How do I extract the downloaded documentation" section https://bugs.python.org/issue35255 opened by deronnax #35257: Add LDFLAGS_NODIST for the LDFLAGS not intended for propagatio
