[issue23903] Generate PC/python3.def by scraping headers

Petr Viktorin added the comment: PEP 652 (bpo-43795) is now in; I'm closing this issue. A cross-platform C parser/checker would still be an improvement, but the constraints (no external dependencies in CPython) and benefits (not many compared to the goal of strict separation of

[issue23903] Generate PC/python3.def by scraping headers

Petr Viktorin added the comment: The symbols exported by python3.dll are now generated from Misc/stable_abi.txt as per PEP 652. See the devguide for more details: https://devguide.python.org/c-api/ This is slightly more work than generating the list fully automatically, but the extra work

[issue23903] Generate PC/python3.def by scraping headers

Petr Viktorin added the comment: I plan to do this slightly differently, see PEP 652. -- nosy: +petr.viktorin ___ Python tracker ___

[issue23903] Generate PC/python3.def by scraping headers

Change by Ned Deily : -- nosy: -ned.deily versions: -Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue23903] Generate PC/python3.def by scraping headers

Change by STINNER Victor : -- components: +C API nosy: +paul.moore ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Reawakening this issue so we can finish it off - the stable ABI has been declared "clean" on python-dev, so there shouldn't be any more concerns about APIs leaking in now. We can assume that everything accessible from the header files should be included in

[issue23903] Generate PC/python3.def by scraping headers

Change by Neil Schemenauer : -- nosy: +nascheme ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue23903] Generate PC/python3.def by scraping headers

Changes by Serhiy Storchaka : -- pull_requests: -1091 ___ Python tracker ___

[issue23903] Generate PC/python3.def by scraping headers

Changes by Donald Stufft : -- pull_requests: +1091 ___ Python tracker ___ ___

[issue23903] Generate PC/python3.def by scraping headers

Roundup Robot added the comment: New changeset 2e5ad97c9c19 by Serhiy Storchaka in branch '3.5': Issue #23903: Added missed Windows-specific names to PC/python3.def. https://hg.python.org/cpython/rev/2e5ad97c9c19 New changeset 86a412584c02 by Serhiy Storchaka in branch '3.6': Issue #23903:

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Unfortunately, there's no way too remove defined names from the DLL during Python 3 at all, except where the prototype was never provided. PyCFunction_New has always been a macro, and PySys_SetDefaultEncoding looks to have been removed before the API was

[issue23903] Generate PC/python3.def by scraping headers

Changes by Serhiy Storchaka : -- dependencies: +Readd PyArg_VaParse to the stable API ___ Python tracker ___

[issue23903] Generate PC/python3.def by scraping headers

Roundup Robot added the comment: New changeset d95fee442e27 by Serhiy Storchaka in branch '3.5': Issue #23903: Added missed names to PC/python3.def. https://hg.python.org/cpython/rev/d95fee442e27 New changeset cb864fc4b3be by Serhiy Storchaka in branch '3.6': Issue #23903: Added missed names to

[issue23903] Generate PC/python3.def by scraping headers

Roundup Robot added the comment: New changeset 8423f86486b3 by Serhiy Storchaka in branch '3.5': Issue #23903: Fixed errors and remove non-existing names in python3.def. https://hg.python.org/cpython/rev/8423f86486b3 New changeset b5470d08969c by Serhiy Storchaka in branch '3.6': Issue #23903:

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: It is not good that the script removes existing names and adds non-existing names. It is safe to remove names that don't exist in all releases starting from 3.2.0. 1. PyCFunction_New shouldn't be removed in maintained releases. This name is exported, but

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Attached a patch that integrates generation into the build of python3.dll. There are also a couple of fixes for headers to retain functions/data that were previously exported. The patch includes an updated python3.def file, though I expect that will go away and

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: They do. Functions PyCFunction_New, _PyTrash_deposit_object etc are compiled and exported even if they don't represented in headers. Actually we can remove the PyCFunction_New function since it was converted to to a macro before Python 3.0 (faabd9867fb8).

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Those macros like PyCFunction_New actually need to become functions again, since we need real exports to forward them to the right place. Converting stable API functions to macros breaks the ABI, since modules compiled against the old one can no longer link to

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: Following patch against 3.6 just adds new names to PC/python3.def. The only removed names are PyExc_MemoryErrorInst and PyUnicode_SetDefaultEncoding. They don't exist in 3.x. -- Added file: http://bugs.python.org/file46027/python3def-3.6.patch

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: Here is a shell script that uses the C preprocessor and other Unix tools to generate PC/python3.def. It doesn't add PyAST_* and PyNode_* functions and doesn't remove PyArg_VaParse* functions. But it still removes some names that are no longer used in public

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: See also issue29058. -- ___ Python tracker ___ ___ Python-bugs-list

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Good call. I think I might actually replace the script with a build step that uses the C preprocessor to get all the names, then a script to generate the file. If someone who knows the POSIX build system can help out, we could generate a list on each build for

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: Reordered to make diffs easier for reviewing. -- Added file: http://bugs.python.org/file46011/python3def.patch ___ Python tracker

[issue23903] Generate PC/python3.def by scraping headers

Roundup Robot added the comment: New changeset 9cb87e53e324 by Serhiy Storchaka in branch '3.5': Sort and remove duplicates from PC/python3.def (issue #23903). https://hg.python.org/cpython/rev/9cb87e53e324 New changeset 0927b5c80c50 by Serhiy Storchaka in branch '3.6': Sort and remove

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: > ast.h and node.h are private headers. You're right, these should be excluded (possibly from the release too?) I've been playing with the script in a separate context and I think I've hit a few issues (though I have made some modifications, so YMMV). It's

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: PyCmpWrapper_Type is not defined. Some underscored names are defined only in special builds (with Py_REF_DEBUG, Py_TRACE_REFS). PyAST_* and PyNode_* functions are not documented, ast.h and node.h are private headers. Names like _PyArg_Parse_SizeT

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: > would this change likely affect any third-party binaries already built for > 3.6.0? On the other hand, if we have to have one, it would be better to have > a compatibility break at 3.6.0 rather than at 3.6.1. Assuming we only make additive changes, then

[issue23903] Generate PC/python3.def by scraping headers

Ned Deily added the comment: I don't know enough about how python3.dll is used in the Windows world to ask the right questions. One obvious one: would this change likely affect any third-party binaries already built for 3.6.0? On the other hand, if we have to have one, it would be better to

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Guess I should at least pose the question - Ned, would you consider the change in my patch for 3.6.0? Only the one file is affected, and it only contributes to one DLL that is (a) recommended, (b) not widely used and (c) very incompatible with the included

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: I just ran into this because PyModule_AddFunctions() was added to the stable ABI in 3.5, but was not added to python3.dll. This is blatantly a compilation error. We _need_ python3.def and the stable ABI to be in sync. I've rerun the script against Python 3.6

[issue23903] Generate PC/python3.def by scraping headers

Zachary Ware added the comment: > There is a code for parsing preprocessor instructions in Argument Clinic. > May be it can be reused here. It already is, actually; its availability was a major factor in my thinking this would be possible and starting to work on it. --

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: See also closely related issue26900. -- ___ Python tracker ___ ___

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: As I recall, the tool is ready to go. The problem is the argument over what to do with all the APIs that currently are incorrect. Zach had a list at PyCon last year, and the consensus seemed to be that "someone" should figure out whether each API should be

[issue23903] Generate PC/python3.def by scraping headers

Serhiy Storchaka added the comment: This tool would be very useful, since new API often accidentally marked as stable API in header files. The script can play just deliberative role, warn if new unstable API is not wrapped with "#ifndef Py_LIMITED_API" or if stable API is not documented. May

[issue23903] Generate PC/python3.def by scraping headers

Zachary Ware added the comment: Unassigning since I haven't made it anywhere with this in a year; if anybody else would like to pick it up, feel free. Otherwise I'll get to it some day :) -- assignee: zach.ware -> versions: +Python 3.6 -Python 3.5

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: It might be nice to have a test that tries to import all of the stable ABI at build time so that the buildbots break if someone adds a new API and doesn't think about what they've done. As an aside, it'd be nice for the script to take the version numbers as args

[issue23903] Generate PC/python3.def by scraping headers

Zachary Ware added the comment: What about including this in Tools/scripts/patchcheck.py? I think that might strike a good enough balance between making changes noticeable and not automatically changing the def file. As far as all the differences this has uncovered, I'll try to make a patch

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: Could the latter break people already using the stable ABI on other platforms? Does it even work for others? I don't know what the equivalent of python3.dll would be. If it doesn't exist outside of Windows, then assuming that everything not in python3.def isn't

[issue23903] Generate PC/python3.def by scraping headers

Martin v. Löwis added the comment: I think the approach taken in this script is incorrect. It leads to false modifications of the stable ABI, making it in fact unstable. Four examples: PyAST_FromNode does not belong to the stable ABI, because struct _node doesn't belong to the stable ABI (see

[issue23903] Generate PC/python3.def by scraping headers

Steve Dower added the comment: That would be the preferred way to do it, but we got no traction from anyone at PyCon or on python-dev when we tried to get people to look at what's changed. I suspect we haven't marketed the stable API well enough to have a lot of users, and the build tools

[issue23903] Generate PC/python3.def by scraping headers

Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23903 ___

[issue23903] Generate PC/python3.def by scraping headers

Zachary Ware added the comment: Same patch, with diff to python3.def to make it available. -- Added file: http://bugs.python.org/file38967/23903_with_python3.def.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23903