[issue11734] Add half-float (16-bit) support to struct module

Changes by Mark Dickinson : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue11734] Add half-float (16-bit) support to struct module

Roundup Robot added the comment: New changeset 519bde9db8e0 by Mark Dickinson in branch 'default': Issue #11734: Add support for IEEE 754 half-precision floats to the struct module. Original patch by Eli Stevens. https://hg.python.org/cpython/rev/519bde9db8e0 -- nosy: +python-dev

[issue11734] Add half-float (16-bit) support to struct module

Raymond Hettinger added the comment: Mark probably already knows this, but I found this SO answer to be informative: http://stackoverflow.com/questions/3886988/how-to-distinguish-different-types-of-nan-float-in-python -- ___ Python tracker

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: I missed Antoine's 2015 comments on the test code. Here's one more version of the patch, which addresses those comments. -- Added file: http://bugs.python.org/file44282/cpython-struct-float16-v9.patch ___ Python

[issue11734] Add half-float (16-bit) support to struct module

Changes by Mark Dickinson : -- versions: +Python 3.6 -Python 3.3 ___ Python tracker ___

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: Updated patch: - Fix NaN packing behaviour. - Expand documentation note on the half-precision format, with links to the wikipedia pages for IEEE 754 and for half-precision. (No direct link to IEEE 754 itself, since it isn't publicly available.) - Use

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: There's some NaN behaviour that needs fixing in the packing code: packing a NaN currently creates a signalling NaN rather than a quiet NaN, and the sign of a NaN isn't respected. (One can make a strong argument that the sign of the NaN doesn't matter, but

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: Thanks, Raymond. I'll do a final pass and aim to apply this in the next couple of days. -- ___ Python tracker ___

[issue11734] Add half-float (16-bit) support to struct module

Changes by Raymond Hettinger : -- assignee: -> mark.dickinson ___ Python tracker ___

[issue11734] Add half-float (16-bit) support to struct module

Raymond Hettinger added the comment: In the docs for note 7, consider expanding the text to describe what a half-float is (1-bit sign, 10 bit significand, 5 bit exponent) and its allowable range. This wasn't necessary for codes "f" and "d" since they have well-known C equivalents, but in

[issue11734] Add half-float (16-bit) support to struct module

Changes by Mark Dickinson : -- assignee: mark.dickinson -> ___ Python tracker ___ ___

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: Refreshed and updated patch, cleaning up some style issues, fixing a failure to return -1 on exception in np_halffloat, and removing some endianness-determining casty expressions in favour of using PY_LITTLE_ENDIAN. -- Added file:

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: Refreshed patch, with a few minor updates: - Use copysign instead of signbit (we don't currently check for signbit in the configure scripts); only check for negative zero when we've already checked equality with 0.0. - Use Py_IS_NAN and Py_IS_INFINITY

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: I don't see how math.isnan or math.isinf would be affected: we're not proposing to make a float16 Python type, so math.isnan would never encounter a float16 value. -- ___ Python tracker rep...@bugs.python.org

[issue11734] Add half-float (16-bit) support to struct module

Paul Hoffman added the comment: Mark: I don't see how math.isnan or math.isinf would be affected: we're not proposing to make a float16 Python type, so math.isnan would never encounter a float16 value. I was assuming that this proposal would also make this part of class 'float' just like

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: Paul: your example works because struct.unpack('f', packed_stuff)[0] returns a regular Python float, backed by C double---i.e., exactly the same type that's returned by struct.unpack('d', packed_stuff)[0]. In effect, the single-precision C value is computed

[issue11734] Add half-float (16-bit) support to struct module

Paul Hoffman added the comment: I *think* what you are saying is that there will be no float16 type, but the result of struct.unpack('eorwhatever', packed_stuff)[0] will be the same as struct.unpack('f', packed_stuff)[0], yes? If so, that's what I wanted. I don't want a new float16; I want

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson added the comment: I'll take a look at the numerics. -- assignee: - mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___

[issue11734] Add half-float (16-bit) support to struct module

Paul Hoffman added the comment: Just another voice for seeing this put in a deployed version of Python. Half-precision floats are becoming more common in applications. Question: does adding this affect math.isnan and math.isinf? -- nosy: +paulehoffman

[issue11734] Add half-float (16-bit) support to struct module

Andrew Svetlov andrew.svet...@gmail.com added the comment: Tests passed, looks good. Code is also looks ok. Documentation is out-of-date. Patch cannot be applied to struct.rst from default branch. Compilation generates warning messages on gcc 4.6.1:

[issue11734] Add half-float (16-bit) support to struct module

Antoine Pitrou pit...@free.fr added the comment: Given than there is no native half float in C (AFAIK), does it make sense to have a native variant of half floats? Also: +{'e', sizeof(short), SHORT_ALIGN,nu_halffloat, np_halffloat}, Shouldn't it be 2 rather than

[issue11734] Add half-float (16-bit) support to struct module

Andrew Svetlov andrew.svet...@gmail.com added the comment: I used half-float in GPU programming and from my perspective it was just native. There are no half-float in C, right. But there are half-floats used in NVIDIA libraries for example and I like to think used format is native and

[issue11734] Add half-float (16-bit) support to struct module

Antoine Pitrou pit...@free.fr added the comment: I used half-float in GPU programming and from my perspective it was just native. There are no half-float in C, right. But there are half-floats used in NVIDIA libraries for example and I like to think used format is native and

[issue11734] Add half-float (16-bit) support to struct module

Andrew Svetlov andrew.svet...@gmail.com added the comment: Antoine, agree with you after explanation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___

[issue11734] Add half-float (16-bit) support to struct module

Changes by Antoine Pitrou pit...@free.fr: -- nosy: +pitrou stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___ ___

[issue11734] Add half-float (16-bit) support to struct module

Changes by Alexander Belopolsky belopol...@users.sourceforge.net: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___ ___

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: I see the distinction about the rounding error now. Thanks for clarifying; I've added more tests as suggested. Looking at _struct.c, line 2085: /* Skip float and double, could be unknown float

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: More questions: The current _PyFloat_Pack4 doesn't round to even: fbits = (unsigned int)(f + 0.5); /* Round */ Is the mismatch going to be a problem? Should I change the _PyFloat_Pack4 implementation while I'm in there? --

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: Made the _PyFloat_Pack2 match the algo in _PyFloat_Pack4, and did similar for Unpack. This should work on platforms that don't have IEEE 754 floats except for situations where an INF or NAN is unpacked (this is the same as the Unpack4

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson dicki...@gmail.com added the comment: There's no disagreement, since they're different cases. [...] What he said. Should I change the _PyFloat_Pack4 implementation while I'm in there? No; let's keep the patch as simple as possible. We can open a separate issue for fixing

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson dicki...@gmail.com added the comment: Is the failure to round-to-even only for legacy formats, or is it for IEEE formats as well? Ah; I see it's just for the non-IEEE formats, so not really an issue. When the float format is known, the code just depends on a (float) cast

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson dicki...@gmail.com added the comment: I'd also suggest adding some more to the test suite here to verify that ties are rounding to the nearest even properly. And I second this suggestion. -- ___ Python tracker

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: All four changes suggested via review from Mark Dickinson have been made. One note: the else if (!(e == 0 f == 0.0)) { change was made; this makes the Pack2 function parallel the Pack4 function slightly less. I agree that it's cleaner

[issue11734] Add half-float (16-bit) support to struct module

Changes by Eli Stevens wickedg...@gmail.com: Removed file: http://bugs.python.org/file21497/cpython-struct-float16-v1.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___

[issue11734] Add half-float (16-bit) support to struct module

Changes by Eli Stevens wickedg...@gmail.com: Removed file: http://bugs.python.org/file21499/cpython-struct-float16-v3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___

[issue11734] Add half-float (16-bit) support to struct module

Mark Wiebe mwwi...@gmail.com added the comment: Just a few small tweaks. I think you meant to subtract the second term here: +#('e', b'\x80\x01', -2.0**-25 + 2.0**-35), # Rounds to minimum subnormal Isn't the x 0 part unnecessary? Maybe a comment explaining why the signbit function is

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: Updated patch with changes suggested by Mark Wiebe. I also removed the commented-out if (!(e == 0 f == 0.0)) part. It feels like the patch is approaching an acceptable state; after both of you agree it's there, what do I need to do next?

[issue11734] Add half-float (16-bit) support to struct module

Mark Dickinson dicki...@gmail.com added the comment: I've only glanced at the patch, but a couple of things: (1) It looks as though the patch assumes that a C double is IEEE 754 binary64 format, and that a C float is IEEE 754 binary32 format. Is that correct? If so, that's a significant

[issue11734] Add half-float (16-bit) support to struct module

Mark Wiebe mwwi...@gmail.com added the comment: The patch currently assumes IEEE 754 with byte-order matching the integer type. I see that pyconfig.h defines three possible cases when IEEE 754 doubles are supported, DOUBLE_IS_LITTLE_ENDIAN_IEEE754, DOUBLE_IS_BIG_ENDIAN_IEEE754, and

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: There seems to be some disagreement about the double-rounding issue; Mark Dickinson posted on python-ideas that he doesn't think that there is one. If possible, I think that removing code paths that aren't needed are generally a good thing,

[issue11734] Add half-float (16-bit) support to struct module

Mark Wiebe mwwi...@gmail.com added the comment: There's no disagreement, since they're different cases. Taking an arbitrary double, rounding to float, then rounding to half definitely has double-rounding issues. (And I would recommend constructing an example to add to the test case to make

[issue11734] Add half-float (16-bit) support to struct module

Changes by Mark Dickinson dicki...@gmail.com: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___ ___

[issue11734] Add half-float (16-bit) support to struct module

Mark Wiebe mwwi...@gmail.com added the comment: Taking a look at the patch, I see you're using the single - half conversion routine from NumPy. This has the double rounding problem when converting double - float - half, so it would be better to use the double - half routine. I implemented it

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: I'm thinking of taking the current float implementation and wrapping it with something like: #if HAS_INT64_TYPE // double implementation goes here #else // float implementation here (what's in the current patch) ... #endif Does

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: I've spelled HAS_INT64_TYPE as follows: +#if SIZEOF_LONG == 8 || SIZEOF_LONG_LONG == 8 +#if SIZEOF_LONG == 8 +typedef unsigned long npy_uint64; +#else +#if SIZEOF_LONG_LONG == 8 +typedef unsigned long long npy_uint64; +#endif +#endif

[issue11734] Add half-float (16-bit) support to struct module

Mark Wiebe mwwi...@gmail.com added the comment: I think this won't work on Windows since there the 64-bit int is generally __int64. If you look at the long long and unsigned long long support in _struct.c you can see a better way to do this: #ifdef HAVE_LONG_LONG and unsigned PY_LONG_LONG

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: Ahh, yes, much cleaner. Thank you. :) I'll remove the previous version. -- Added file: http://bugs.python.org/file21499/cpython-struct-float16-v3.patch ___ Python tracker rep...@bugs.python.org

[issue11734] Add half-float (16-bit) support to struct module

Changes by Eli Stevens wickedg...@gmail.com: Removed file: http://bugs.python.org/file21498/cpython-struct-float16-v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11734 ___

[issue11734] Add half-float (16-bit) support to struct module

New submission from Eli Stevens wickedg...@gmail.com: Numpy 1.6.0 adds support for a half-float (16-bit) data type, but cannot currently export a buffer interface to the data since neither PEP 3118 nor the struct module (referenced by PEP 3118) support the data type. I am proposing that the

[issue11734] Add half-float (16-bit) support to struct module

Eli Stevens wickedg...@gmail.com added the comment: Initial patch; tests pass. A couple notes: - I manually excised some commented changes to structmodule.{h,c} from the .patch; once I get a determination on if those files need to be updated or not I will discard the changes or implement