[issue19904] Add 128-bit integer support to struct

[Joannah Nanjekye]

Change by Joannah Nanjekye :


--
nosy: +nanjekyejoannah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Zachary Ware]

Changes by Zachary Ware zachary.w...@gmail.com:


--
stage:  - needs patch
versions: +Python 3.6 -Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Stefan Krah]

Changes by Stefan Krah stefan-use...@bytereef.org:


--
nosy:  -skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

So where do we go from here - is there enough support for this to proceed, or 
vote the idea off the island :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Gregory P. Smith]

Gregory P. Smith added the comment:

Pick two suitable letters for the int128 and uint128 cases and create a patch 
for an implementation with tests (targeting Python 3.5).  The implementation 
needs to compile and work even when the platform C compiler Python is compiled 
with does not have a 128-bit integer type (so the struct module always supports 
this rather than depending upon the platform native support in C).

I wouldn't bother with 256 and 512 bit integer support because I've never seen 
those used.  If that changes before 3.5 is released (a couple years), they can 
be added.

--
nosy: +gregory.p.smith

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Francisco Martín Brugué]

Francisco Martín Brugué added the comment:

 If performance is the reason for the feature: My impression is that
 the goal of the struct module is not necessarily top performance.

I'm not sure if it applies but on #19905 (message 205345 [1])
is said its a dependency for that issue


[1] http://bugs.python.org/issue19905#msg205345

--
nosy: +francismb

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Meador Inge]

Changes by Meador Inge mead...@gmail.com:


--
nosy: +meador.inge

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

Stefan, performance is not the principle motivator here: the intention is that 
it is just sensible to support this integer type, just like supporting int64 
since it is an intrinsic type supported by CPU's.

Of course any integer length could be handled by memoryview unpacker, but this 
would not really make sense for int64. My view is that any intrinsic type (ie. 
register type) is the key point at which it should be supported by struct, and 
not a custom unpacker. 128/256/512 bit integers are in this category..

Principally this is so that it can be used by ctypes (see #19905).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Of course any integer length could be handled by memoryview unpacker,
 but this would not really make sense for int64. My view is that any
 intrinsic type (ie. register type) is the key point at which it should
 be supported by struct, and not a custom unpacker. 128/256/512 bit
 integers are in this category..

I don't think register types matter here. The struct module provides
interoperability with low-level *data types* (in C and other common
languages), not CPU *registers*.

So IMHO the question is whether there is an use case for 128-bit
integers; *not* for arrays of four 32-bit integers packed in a 128-bit
register.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

Antoine,

No, the SIMD vector should be treated as a int128 (for eg.), not as 4 python 
ints. It is the unpacking process that unpacks into 4 python ints. The two 
shouldn't be confused - you definitely want to be able to treat the vector in 
both states (packed and unpacked).

I'm not saying that Python should necessarily make use of SIMD instructions 
(don't think performance is critical there in py), but that it should at least 
be access to a raw 128/256/512 bit integer within a C structure. Part of this 
is interacting with SIMD'able structures - at least in my use cases :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Antoine Pitrou]

Antoine Pitrou added the comment:

 The two shouldn't be confused - you definitely want to be able to
 treat the vector in both states (packed and unpacked).

Why do you need the packed state as a 128-bit int, rather than as a
bytes object? You are not doing arithmetic on it, AFAIU.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

Antoine,

 I don't think register types matter here. The struct module provides
 interoperability with low-level *data types* (in C and other common
 languages), not CPU *registers*.

OK, see where you're coming from. I guess my thinking was such that struct (and 
ctypes) support is required for data types that are common within C structures. 
The fact that 128-bit ints are now appearing within these structures is due to 
the fact that CPU support has dramatically improved, and are supported natively 
in register types. So it was kind of an indirect connection.

So the question I guess is whether you think these types are common enough in 
C structs, which for me they certainly are for my data processing 
applications.. (pretty please :)

 So IMHO the question is whether there is an use case for 128-bit
 integers; *not* for arrays of four 32-bit integers packed in a 128-bit
 register.

Agree completely - just stick to thinking about atomic 128-bit integers. A 
pack/unpack function should do the conversion, and have nothing to do with this 
struct support.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Stefan Krah]

Stefan Krah added the comment:

If performance is the reason for the feature: My impression is that
the goal of the struct module is not necessarily top performance.

E.g. in memoryview the custom unpackers for comparisons are 30-60 times
faster than using the struct module.

--
nosy: +skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Mark Dickinson]

Mark Dickinson added the comment:

Do you have a particular application / use-case in mind?  Would int.from_bytes 
and int.to_bytes fill the need?

--
nosy: +mark.dickinson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Raymond Hettinger]

Changes by Raymond Hettinger raymond.hettin...@gmail.com:


--
versions:  -Python 2.7, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

The use case is interacting with C structures that have 128/256/512 bit 
integers in them. These require correct memory alignment, and can't reliably be 
hacked with multiple int64's.

These size ints come from the fact that CPU's now have registers of these 
sizes, and can't be serviced with int.from/to_bytes for performance reasons. 
The same reason int64 being supported with this approach would be very 
inconvenient and terrible for performance since they are an intrinsic type in 
modern hardware, not a software construction.

Two key use cases I can think of are:

- any form of integer SIMD operation (vectors)
- hosting and maintaining hash values which are routinely 128-bit and greater

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

Fil Mackay added the comment:

I noticed that python 2.7 has been removed - why would this not be appropriate 
for this version?

Given the current level of use of this version, I see it's inclusion as very 
important - without having to be relegated to 3.x only.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Alex Gaynor]

Changes by Alex Gaynor alex.gay...@gmail.com:


--
nosy: +alex

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Tim Peters]

Tim Peters added the comment:

Fil, here's the release schedule for Python 2.8:

http://www.python.org/dev/peps/pep-0404/

In short, 2.8 will never be released (well, never by us), and only bugfixes can 
be applied to the 2.7 line.  That's why 2.7 was removed.  Regardless of its 
merits, this is a new feature request, not a bugfix.

A future Python 3 release is the only possibility for new features.

--
nosy: +tim.peters

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Antoine Pitrou]

Antoine Pitrou added the comment:

 any form of integer SIMD operation (vectors)

Wouldn't these be appropriately represented by a tuple of integers (or floats)? 
For example, a SIMD vector of four 32-bit integers could be represented as four 
Python ints. Or would that be terrible for performance?

Note that the struct module may include support for int128_t, but it certainly 
won't have native support for every SIMD vector format under the sun (if they 
have different alignment requirements).

 hosting and maintaining hash values which are routinely 128-bit and greater

That sounds like a job for a bytes object (as returned by e.g. 
hashlib.sha1(...).digest()).

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19904] Add 128-bit integer support to struct

[Fil Mackay]

New submission from Fil Mackay:

I've been looking at adding 128-bit support to the struct module. Currently 
only named integer types are supported, which vary in implementation. These 
include:

short
int
long
long long

Depending on the platform, none may translate to 128-bit integer (the case with 
all platforms today?).

One approach would be to make a new type that relates specifically to 128-bit 
integer, side-stepping the naming approaches to integer in C.

The other, would be to make new types for all integer sizes that relate to 
specific sizes, instead of relying on C namings. Much bigger implications?

I propose creating new types:

o: __int128_t
O: __uint128_t
t: __int256_t (why not?)
T: __uint256_t
v: __int512_t (what, too far?)
V: __int512_t

What implications are there here in killing the connection between a C named 
type and a specific size?

--
components: ctypes
messages: 205344
nosy: fil
priority: normal
severity: normal
status: open
title: Add 128-bit integer support to struct
type: enhancement
versions: Python 2.7, Python 3.4, Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19904
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com