[issue26746] struct.pack(): trailing padding bytes on x64

[Stefan Krah]

Change by Stefan Krah :


--
nosy: +Eric.Wieser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Stefan Krah]

Change by Stefan Krah :


--
versions: +Python 3.7 -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Stefan Krah]

Stefan Krah  added the comment:

I have just worked on PEP-3118 ==> Datashape translation and I have
encountered many issues similar to the ones in the PR referenced by
Allan.

It seems to me that we should simplify the PEP-3118 struct syntax as much 
as possible in order to remove any ambiguities.


I think generally that numpy's approach is the best for data interchange, so I
would propose this modified struct syntax for PEP-3118:


1) Padding is always explicit and exact, also for natively aligned types.

2) Padding is only allowed in struct fields.

3) Trailing padding is explicit.

4) If no padding is present in a struct, it is assumed to be packed with
alignment 1 for the entire struct.

5) The tuple syntax "bxL" is not supported, only the T{} syntax with
explicit field names.

6) Repetition "10s" is only allowed for bytes. "10f" is a tuple (not
supported), an array of 10 floats would be (10)f.

7) Modifiers (@, =, <, >, !) are only given for primitive data types,
not for entire structs or types.

8) Implementations are free to reject any padding that would not arise
naturally by specifying alignment or packing constraints (like gcc does 
with attributes).


Here is my implementation with a grammar:

  https://github.com/plures/ndtypes/blob/master/libndtypes/compat/bpgrammar.y

Some tests against numpy:

  https://github.com/plures/xnd/blob/master/python/test_xnd.py#L1509


I think the best way forward would be to tweak the above grammar so that
it covers everything that numpy can export.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Allan Haldane]

Allan Haldane added the comment:

Hello,

Over at numpy I have a proposed fix for the bug you discovered, that numpy 
drops trailing padding in the 3118 format string. My strategy is going to make 
numpy interpret format strings exactly the same way as the struct module, let 
me know if you disagree.

See https://github.com/numpy/numpy/pull/7798

--
nosy: +Allan Haldane

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Meador Inge]

Meador Inge added the comment:

I'm not to crazy about the trailing padding syntax either.  The behavior is 
documented all the way back to Python 2.6.  So, I would be hesitant to change 
it now.

If the new 'T{...}' struct syntax from issue3132 ever gets added, then maybe we 
could address this there?

FWIW, internal and trailing padding is implementation defined by the C 
standard.  That being said, most compilers I have worked with add the trailing 
padding.

--
nosy: +meador.inge

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Stefan Krah]

Stefan Krah added the comment:

Thank you.  So technically, in the above NumPy example the format
string generated by NumPy would be considered incomplete if we assume
struct syntax:

>>> m = memoryview(x)
>>> m.format
'T{B:x:xxxL:y:B:z:}'


I find this "0L" thing a very odd notation. Taking care of this
manually requires a) knowledge of what the compiler does and b)
searching for the largest struct member.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Martin Panter]

Martin Panter added the comment:

This behaviour seems to be documented, although it is not very explicit, and a 
bit surprising to me. See the third note at the end of 
: 
“align the end . . . with a repeat count of zero”, and the example

>>> pack('llh0l', 1, 2, 3)
b'\x00\x00\x00\x01\x00\x00\x00\x02\x00\x03\x00\x00'

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26746] struct.pack(): trailing padding bytes on x64

[Stefan Krah]

New submission from Stefan Krah:

On the x64 architecture gcc adds trailing padding bytes after the last
struct member.  NumPy does the same:

>>> import numpy as np
>>> 
>>> t = np.dtype([('x', 'u1'), ('y', 'u8'), ('z', 'u1')], align=True)
>>> x = np.array([(1, 2, 3)], dtype=t)
>>> x.tostring()
b'\x01\xf7\xba\xab\x03\x7f\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'


The struct module in native mode does not:

>>> struct.pack("BQB", 1, 2, 3)
b'\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03'


I'm not sure if this is intended -- or if full compatibility to
native compilers is even achievable in the general case.

--
components: Extension Modules
messages: 263315
nosy: mark.dickinson, skrah
priority: normal
severity: normal
status: open
title: struct.pack(): trailing padding bytes on x64
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com