Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Fri, Nov 9, 2018 at 4:33 PM Victor Stinner wrote: > It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the > name) to get the new API. The current C API is unchanged. > While one can hope that this will be the only time the C API will be revised, it may be better to number it instead of calling it "NEW". 20 years from now, it won't feel new anymore. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
To hide all implementation details, I propose to stop using macros and use function calls instead. For example, replace: #define PyTuple_GET_ITEM(op, i) \ (((PyTupleObject *)(op))->ob_item[i]) with: # define PyTuple_GET_ITEM(op, i) PyTuple_GetItem(op, i) With this change, C extensions using PyTuple_GET_ITEM() does no longer dereference PyObject* nor access PyTupleObject.ob_item. For example, PyPy doesn't have to convert all tuple items to PyObject, but only create one PyObject for the requested item. Another example is that it becomes possible to use a "CPython debug runtime" which checks at runtime that the first argument is a tuple and that the index is valid. For a longer explanation, see the idea of different "Python runtimes": https://pythoncapi.readthedocs.io/runtimes.html Replacing macros with function calls is only a first step. It doesn't solve the problem of borrowed references for example. Obviously, such change has a cost on performances. Sadly, I didn't run a benchmark yet. At this point, I mostly care about correctness and the feasibility of the whole project. I also hope that the new C API will allow to implement new optimizations which cannot even be imagined today, because of the backward compatibility. The question is if the performance balance is positive or not at the all :-) Hopefully, there is no urgency to take any decision at this point. The whole project is experimental and can be cancelled anytime. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
> > That's exactly why I dislike "New", it's like adding "Ex" or "2" to a > function name :-) > > Well, before bikeshedding on the C define name, I would prefer to see > if the overall idea of trying to push code for the new C API in the > master branch is a good idea, or if it's too early and the experiment > must continue in a fork. Rather than adding yet another pre-processor directive for this I would suggest just adding a new header file that only has the new stable API. For example it could just be "py.h" or "pyapi.h". It would have all of the definitions for the stable API. While that would involve some duplication from the existing headers, I don't think it would be such a big deal - the idea is the API won't change, methods won't be removed, and occasionally new methods will get added in a very thoughtful manner. Having it be separate will force thought and conversation about it. It would also make it very easy to look and see what exactly is in the stable API as well. There's be a pretty flat list which can be consulted, and hopefully it ends up not being super huge either. BTW, thanks for continuing to push on this Victor, it seems like great progress! On Fri, Nov 9, 2018 at 4:57 PM Victor Stinner wrote: > Le sam. 10 nov. 2018 à 01:49, Michael Selik a écrit : > >> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the > >> name) to get the new API. The current C API is unchanged. > > > > While one can hope that this will be the only time the C API will be > revised, it may be better to number it instead of calling it "NEW". 20 > years from now, it won't feel new anymore. > > That's exactly why I dislike "New", it's like adding "Ex" or "2" to a > function name :-) > > Well, before bikeshedding on the C define name, I would prefer to see > if the overall idea of trying to push code for the new C API in the > master branch is a good idea, or if it's too early and the experiment > must continue in a fork. > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dinoviehland%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Fri, Nov 9, 2018 at 6:03 PM, Victor Stinner wrote:
> Le sam. 10 nov. 2018 à 02:50, Nathaniel Smith a écrit :
>> Doesn't this mean that you're just making the C API larger and more
>> complicated, rather than simplifying it? You cite some benefits
>> (tagged pointers, changing the layout of PyObject, making PyPy's life
>> easier), but I don't see how you can do any of those things so long as
>> the current C API remains supported.
>
> Tagged pointers and changing the layout of PyObject can only be
> experimented in a new different Python runtime which only supports C
> extensions compiled with the new C API. Technically, it can be CPython
> compiled with a different flag, as there is already python3-dbg (debug
> mode, ./configure --with-pydebug) and python3 (release mode). Or it
> can be CPython fork.
>
> I don't propose to experiment tagged pointer or changing the layout of
> PyObject in CPython. It may require too many changes and it's unclear
> if it's worth it or not. I only propose to implement the least
> controversial part of the new C API in the master branch, since
> maintaining this new C API in a fork is painful.
>
> I cannot promise that it will make PyPy's life easier. PyPy developers
> already told me that they already implemented the support of the
> current C API. The promise is that if you use the new C API, PyPy
> should be more efficient, because it would have less things to
> emulate. To be honest, I'm not sure at this point, I don't know PyPy
> internals. I also know that PyPy developers always complain when we
> *add new functions* to the C API, and there is a non-zero risk that I
> would like to add new functions, since current ones have issues :-) I
> am working with PyPy to involve them in the new C API.
So is it fair to say that your plan is that CPython will always use
the current ("old") API internally, and the "new" API will be
essentially an abstraction layer, that's designed to let people write
C extensions that target the old API, while also being flexible enough
to target PyPy and other "new different Python runtimes"?
If so, then would it make more sense to develop this as an actual
separate abstraction layer? That would have the huge advantage that it
could be distributed and versioned separately from CPython, different
packages could use different versions of the abstraction layer, PyPy
isn't forced to immediately add a bunch of new APIs...
-n
--
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Le sam. 10 nov. 2018 à 01:49, Michael Selik a écrit : >> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the >> name) to get the new API. The current C API is unchanged. > > While one can hope that this will be the only time the C API will be revised, > it may be better to number it instead of calling it "NEW". 20 years from now, > it won't feel new anymore. That's exactly why I dislike "New", it's like adding "Ex" or "2" to a function name :-) Well, before bikeshedding on the C define name, I would prefer to see if the overall idea of trying to push code for the new C API in the master branch is a good idea, or if it's too early and the experiment must continue in a fork. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Sometimes, code is easier to understand than a long explanation, so here is a very simple example of modified function for the new C API: https://bugs.python.org/issue35206 https://github.com/python/cpython/pull/10443/files PyTuple_GET_ITEM() becomes a function call and the function implementation checks arguments at runtime if compiled in debug mode. Technically, the header file still uses a macro, to implicitly cast to PyObject*, since currently the macro accepts any type, and the new C API should not change that. Victor Le sam. 10 nov. 2018 à 01:53, Victor Stinner a écrit : > > To hide all implementation details, I propose to stop using macros and > use function calls instead. For example, replace: > > #define PyTuple_GET_ITEM(op, i) \ >(((PyTupleObject *)(op))->ob_item[i]) > > with: > > # define PyTuple_GET_ITEM(op, i) PyTuple_GetItem(op, i) > > With this change, C extensions using PyTuple_GET_ITEM() does no longer > dereference PyObject* nor access PyTupleObject.ob_item. For example, > PyPy doesn't have to convert all tuple items to PyObject, but only > create one PyObject for the requested item. Another example is that it > becomes possible to use a "CPython debug runtime" which checks at > runtime that the first argument is a tuple and that the index is > valid. For a longer explanation, see the idea of different "Python > runtimes": > >https://pythoncapi.readthedocs.io/runtimes.html > > Replacing macros with function calls is only a first step. It doesn't > solve the problem of borrowed references for example. > > Obviously, such change has a cost on performances. Sadly, I didn't run > a benchmark yet. At this point, I mostly care about correctness and > the feasibility of the whole project. I also hope that the new C API > will allow to implement new optimizations which cannot even be > imagined today, because of the backward compatibility. The question is > if the performance balance is positive or not at the all :-) > Hopefully, there is no urgency to take any decision at this point. The > whole project is experimental and can be cancelled anytime. > > Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Le sam. 10 nov. 2018 à 02:50, Nathaniel Smith a écrit : > Doesn't this mean that you're just making the C API larger and more > complicated, rather than simplifying it? You cite some benefits > (tagged pointers, changing the layout of PyObject, making PyPy's life > easier), but I don't see how you can do any of those things so long as > the current C API remains supported. Tagged pointers and changing the layout of PyObject can only be experimented in a new different Python runtime which only supports C extensions compiled with the new C API. Technically, it can be CPython compiled with a different flag, as there is already python3-dbg (debug mode, ./configure --with-pydebug) and python3 (release mode). Or it can be CPython fork. I don't propose to experiment tagged pointer or changing the layout of PyObject in CPython. It may require too many changes and it's unclear if it's worth it or not. I only propose to implement the least controversial part of the new C API in the master branch, since maintaining this new C API in a fork is painful. I cannot promise that it will make PyPy's life easier. PyPy developers already told me that they already implemented the support of the current C API. The promise is that if you use the new C API, PyPy should be more efficient, because it would have less things to emulate. To be honest, I'm not sure at this point, I don't know PyPy internals. I also know that PyPy developers always complain when we *add new functions* to the C API, and there is a non-zero risk that I would like to add new functions, since current ones have issues :-) I am working with PyPy to involve them in the new C API. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Fri, Nov 9, 2018 at 4:30 PM, Victor Stinner wrote: > Ah, important points. I don't want to touch the current C API nor make > it less efficient. And compatibility in both directions (current C API > <=> new C API) is very important for me. There is no such plan as > "Python 4" which would break the world and *force* everybody to > upgrade to the new C API, or stay to Python 3 forever. No. The new C > API must be an opt-in option, and current C API remains the default > and not be changed. Doesn't this mean that you're just making the C API larger and more complicated, rather than simplifying it? You cite some benefits (tagged pointers, changing the layout of PyObject, making PyPy's life easier), but I don't see how you can do any of those things so long as the current C API remains supported. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signalling NANs
works for me, too: In [9]: x = cast_int2float(0x7ff80001) In [10]: hex(cast_float2int(x)) Out[10]: '0x7ff80001' In [11]: x = cast_int2float(0x7ff1) In [12]: hex(cast_float2int(x)) Out[12]: '0x7ff1' OS-X, conda build: Python 3.7.0 | packaged by conda-forge | (default, Aug 27 2018, 17:24:52) [Clang 6.1.0 (clang-602.0.53)] on darwin I suspect it depends on the compiler's math library But neither is raising an exception: In [3]: x = cast_int2float(0x7ff1) In [4]: y = cast_int2float(0x7ff80001) In [5]: 2.0 / x Out[5]: nan In [6]: 2.0 / y Out[6]: nan When should it? -CHB On Fri, Nov 9, 2018 at 3:46 AM, Serhiy Storchaka wrote: > 09.11.18 13:05, Steven D'Aprano пише: > >> py> x = cast_int2float(0x7ff1) >> py> x >> nan >> py> hex(cast_float2int(x)) >> '0x7ff80001' >> > > I got '0x7ff1'. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris. > barker%40noaa.gov > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Hi, The current C API of Python is both a strength and a weakness of the Python ecosystem as a whole. It's a strength because it allows to quickly reuse a huge number of existing libraries by writing a glue for them. It made numpy possible and this project is a big sucess! It's a weakness because of its cost on the maintenance, it prevents optimizations, and more generally it prevents to experiment modifying Python internals. For example, CPython cannot use tagged pointers, because the existing C API is heavily based on the ability to dereference a PyObject* object and access directly members of objects (like PyTupleObject). For example, Py_INCREF() modifies *directly* PyObject.ob_refcnt. It's not possible neither to use a Python compiled in debug mode on C extensions (compiled in release mode), because the ABI is different in debug mode. As a consequence, nobody uses the debug mode, whereas it is very helpful to develop C extensions and investigate bugs. I also consider that the C API gives too much work to PyPy (for their "cpyext" module). A better C API (not leaking implementation) details would make PyPy more efficient (and simplify its implementation in the long term, when the support for the old C API can be removed). For example, PyList_GetItem(list, 0) currently converts all items of the list to PyObject* in PyPy, it can waste memory if only the first item of the list is needed. PyPy has much more efficient storage than an array of PyObject* for lists. I wrote a website to explain all these issues with much more details: https://pythoncapi.readthedocs.io/ I identified "bad APIs" like using borrowed references or giving access to PyObject** (ex: PySequence_Fast_ITEMS). I already wrote an (incomplete) implementation of a new C API which doesn't leak implementation details: https://github.com/pythoncapi/pythoncapi It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the name) to get the new API. The current C API is unchanged. Ah, important points. I don't want to touch the current C API nor make it less efficient. And compatibility in both directions (current C API <=> new C API) is very important for me. There is no such plan as "Python 4" which would break the world and *force* everybody to upgrade to the new C API, or stay to Python 3 forever. No. The new C API must be an opt-in option, and current C API remains the default and not be changed. I have different ideas for the compatibility part, but I'm not sure of what are the best options yet. My short term for the new C API would be to ease the experimentation of projects like tagged pointers. Currently, I have to maintain the implementation of a new C API which is not really convenient. -- Today I tried to abuse the Py_DEBUG define for the new C API, but it seems to be a bad idea: https://github.com/python/cpython/pull/10435 A *new* define is needed to opt-in for the new C API. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Signalling NANs
I'm trying to understand some unexpected behaviour with float NANs in
Python 3.5.
Background: in IEEE-754 maths, NANs (Not A Number) come in two flavours,
so called "quiet NANs" and "signalling NANs". By default, arithmetic
operations on qnans return a qnan; operations on snans "signal", which
in Python terms means raising an exception.
The original IEEE-754 standard didn't specify how to distinguish a qnan
from a snan, but a de facto standard arose that bit 51 of the float was
the "quiet bit", if it were set, the NAN was quiet.
For the purposes of this email, I'm going to assume that standard is in
place, even though technically speaking it is platform-dependent.
According to my tests, it seems that we cannot create snans in Python.
The float constructor doesn't recognise "snan", raising ValueError. Nor
can we convert a Decimal snan into a float:
py> from decimal import Decimal
py> snan = Decimal('snan')
py> float(dsnan)
Traceback (most recent call last):
File "", line 1, in
ValueError: cannot convert signaling NaN to float
But unexpectedly (to me at least), we apparently cannot even create a
signalling NAN by casting 64 bits to a float. Here are the functions I
use to do the cast:
from struct import pack, unpack
def cast_float2int(x):
return unpack(' x = cast_int2float(0x7ff80001)
py> x
nan
py> hex(cast_float2int(x))
'0x7ff80001'
So far so good. But now let me try with a signalling NAN:
py> x = cast_int2float(0x7ff1)
py> x
nan
py> hex(cast_float2int(x))
'0x7ff80001'
So it seems that the "quiet" bit is automatically set, even when using
the struct module, making it impossible to create snan floats.
Is this intended? If so, why?
Is this meant as a language feature?
Thanks in advance,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signalling NANs
09.11.18 13:05, Steven D'Aprano пише: py> x = cast_int2float(0x7ff1) py> x nan py> hex(cast_float2int(x)) '0x7ff80001' I got '0x7ff1'. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2018-11-02 - 2018-11-09) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open6834 ( +6) closed 40118 (+47) total 46952 (+53) Open issues with patches: 2726 Issues opened (42) == #10536: Enhancements to gettext docs https://bugs.python.org/issue10536 reopened by serhiy.storchaka #32409: venv activate.bat is UTF-8 encoded but uses current console co https://bugs.python.org/issue32409 reopened by pablogsal #33486: regen autotools related files https://bugs.python.org/issue33486 reopened by ned.deily #34011: Default preference not given to venv DLL's https://bugs.python.org/issue34011 reopened by pablogsal #35145: sqlite3: "select *" should optionally sniff and autoconvert TE https://bugs.python.org/issue35145 reopened by matrixise #35149: pip3 show causing Error for ConfigParaser https://bugs.python.org/issue35149 opened by mdileep #35151: Python 2 xml.etree.ElementTree documentation tutorial uses und https://bugs.python.org/issue35151 opened by epakai #35153: Allow to set headers in xmlrpc.client.ServerProxy https://bugs.python.org/issue35153 opened by ced #35155: Clarify Protocol Handlers in urllib.request Docs https://bugs.python.org/issue35155 opened by Denton-L #35156: Consider revising documentation on Python Builds from source https://bugs.python.org/issue35156 opened by neyuru #35157: Missing pyconfig.h when building from source and pgo flag is e https://bugs.python.org/issue35157 opened by neyuru #35158: Fix PEP 3115 to NOT imply that the class dictionary is used in https://bugs.python.org/issue35158 opened by joydiamond #35163: locale: setlocale(..., 'eo') sets non-existing locale https://bugs.python.org/issue35163 opened by carmenbianca #35164: socket.getfqdn and socket.gethostbyname fail on MacOS https://bugs.python.org/issue35164 opened by ssbarnea #35165: Possible wrong method name in attribute references doc https://bugs.python.org/issue35165 opened by denis-osipov #35166: BUILD_MAP_UNPACK doesn't function as expected for dict subclas https://bugs.python.org/issue35166 opened by bup #35168: shlex punctuation_chars inconsistency https://bugs.python.org/issue35168 opened by tphh #35169: Improve error messages for assignment https://bugs.python.org/issue35169 opened by serhiy.storchaka #35172: Add support for other MSVC compiler versions to distutils. dis https://bugs.python.org/issue35172 opened by Ali Rizvi-Santiago #35173: Re-use already existing functionality to allow Python 2.7.x (b https://bugs.python.org/issue35173 opened by Ali Rizvi-Santiago #35174: Calling for super().__str__ seems to call self.__repr__ in lis https://bugs.python.org/issue35174 opened by Camion #35177: Add missing dependencies between AST/parser header files https://bugs.python.org/issue35177 opened by vstinner #35178: Typo/trivial mistake in warnings.py (may be related to 2.x to https://bugs.python.org/issue35178 opened by tashrifbillah #35181: Doc: Namespace Packages: Inconsistent documentation of __loade https://bugs.python.org/issue35181 opened by mdk #35182: Popen.communicate() breaks when child closes its side of pipe https://bugs.python.org/issue35182 opened by and800 #35183: os.path.splitext documentation needs typical example https://bugs.python.org/issue35183 opened by shaungriffith #35184: Makefile is not correctly generated when compiling pyextat wit https://bugs.python.org/issue35184 opened by mgmacias95 #35185: Logger race condition - loses lines if removeHandler called fr https://bugs.python.org/issue35185 opened by benspiller #35186: distutils.command.upload uses deprecated platform.dist with bd https://bugs.python.org/issue35186 opened by p-ganssle #35189: PEP 475: fnctl functions are not retried if interrupted by a s https://bugs.python.org/issue35189 opened by akeskimo #35190: collections.abc.Sequence cannot be used to test whether a clas https://bugs.python.org/issue35190 opened by hroncok #35191: socket.setblocking(x) treats multiples of 2**32 as False https://bugs.python.org/issue35191 opened by izbyshev #35192: pathlib mkdir throws FileExistsError when not supposed to https://bugs.python.org/issue35192 opened by Adam Dunlap #35193: Off by one error in peephole call to find_op on case RETURN_VA https://bugs.python.org/issue35193 opened by gregory.p.smith #35194: A typo in a constant in cp932 codec https://bugs.python.org/issue35194 opened by izbyshev #35195: Pandas read_csv() is 3.5X Slower on Python 3.7.1 vs Python 3.6 https://bugs.python.org/issue35195 opened by Dragoljub #35196: IDLE text squeezer is too aggressive and is slow https://bugs.python.org/issue35196 opened by rhettinger #35197: graminit.h defines very generic names like 'stmt' or 'test' https://bugs.python.org/issue35197 opened by vstinner #35198: Build issue while compiling cpp
