[issue3132] implement PEP 3118 struct changes

Changes by Stefan Krah : -- versions: +Python 3.6 -Python 3.3 ___ Python tracker ___ ___

[issue3132] implement PEP 3118 struct changes

Stefan Krah added the comment: Here's a grammar that roughly describes the subset that NumPy supports. As for implementing this in the struct module: There is a new data description language on the horizon: http://datashape.readthedocs.org/en/latest/ It does not have all the low-level

[issue3132] implement PEP 3118 struct changes

Changes by Stefan Krah stefan-use...@bytereef.org: -- nosy: -skrah ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list

[issue3132] implement PEP 3118 struct changes

Paul Hoffman added the comment: Following up on http://mail.python.org/pipermail/python-ideas/2011-March/009656.html, I would like to request that struct also handle half-precision floats directly. It's a short change, and half-precision floats are becoming much more popular in applications.

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Paul: there's already an open issue for adding float16 to the struct module: see issue 11734. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Paul Hoffman added the comment: Whoops, never mind. Thanks for the pointer to 11734. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___

[issue3132] implement PEP 3118 struct changes

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list

[issue3132] implement PEP 3118 struct changes

Changes by Alexander Belopolsky alexander.belopol...@gmail.com: -- nosy: +belopolsky -Alexander.Belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Nick Coghlan added the comment: Following up here after rejecting #15622 as invalid The unicode codes in PEP 3118 need to be seriously rethought before any related changes are made in the struct module. 1. The 'c' and 's' codes are currently used for raw bytes data (represented as bytes

[issue3132] implement PEP 3118 struct changes

Stefan Krah stefan-use...@bytereef.org added the comment: Yes, definitely. I'm going to push a new memoryview implementation (complete for all 1D/native format cases) in a couple of days. Once that is done, perhaps we could create a memoryview-struct branch on top of that. -- nosy:

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Is this work something that might be suitable for the features/pep-3118 repo (http://hg.python.org/features/pep-3118/) ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Is there still any interest in this work? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Yes, there's interest (at least here). I've just been really short on Python-time recently, so haven't found time to review your patch. -- ___ Python tracker rep...@bugs.python.org

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: I'm going to unassign for now; I still hope to look at this at some point, but can't see a time in the near future when it's going to happen. -- assignee: mark.dickinson - ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Changes by Antoine Pitrou pit...@free.fr: -- assignee: meador.inge - mark.dickinson stage: needs patch - patch review versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Attached is the latest version of the struct string patch. I tested on OS X 10.6.5 (64-bit) and Ubuntu 10.04 (32-bit). I also scanned for memory problems with Valgrind. There is one test failing on 32-bit systems ('test_crasher'). This is

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Another possibility is to implement the 'O' format unsafely [...] Hmm. I don't much like that idea. Historically, it's supposed to be very difficult to segfault the Python interpreter with pure Python code (well except if you're using

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: For reference, Numpy's PEP 3118 implementation is here: Thanks for that, and the other information you give; that's helpful. It sounds like we're on the same page with respect to alignment of substructs. (Bar the mostly academic question

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: For reference, Numpy's PEP 3118 implementation is here BTW, does this already exist in a released version of NumPy? If not, when is it likely to appear in the wild? -- ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Pauli Virtanen p...@iki.fi added the comment: I still like the idea of scoped endianness markers in the substructs, but if we have to abandon that for compatibility with NumPy that's okay. That, or change the Numpy implementation. I don't believe there's yet much code in the wild that

[issue3132] implement PEP 3118 struct changes

Pauli Virtanen p...@iki.fi added the comment: For reference, Numpy's PEP 3118 implementation is here: http://github.com/numpy/numpy/blob/master/numpy/core/_internal.py#L357 http://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/buffer.c#L76 It would be a good idea to ensure that

[issue3132] implement PEP 3118 struct changes

Changes by Meador Inge mead...@gmail.com: -- assignee: teoliphant - meador.inge priority: critical - high stage: unit test needed - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Travis Oliphant oliph...@enthought.com added the comment: On May 19, 2010, at 2:38 PM, Mark Dickinson wrote: Mark Dickinson dicki...@gmail.com added the comment: Travis, this issue is still assigned to you. Do you plan to work on this at some stage, or may I unassign you? You may

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: is that correct, or should the production list be something like: Yup, you are right. I will change the grammar. Whether these cases are valid or not (personally, I think they should be), we should add some tests for them. '' *is*

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou pit...@free.fr added the comment: The possibility of mixing native size/alignment with standard size/alignment in a single format string makes me a bit uneasy I agree. It is hard for me to see how this might be used. Without having anything more constructive to add, I

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Thanks for the new patch. ... If un-aligned, native data-types are requested, then the endian specification is '^'. However, I am not quite sure how to interpret the last sentence. Hmm. Seems like the PEP authors are proposing a new

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: After a bit more thought (and after soliciting a couple of opinions on #python-dev), I'm convinced that endianness changes within a substruct should be local to that substruct: - it makes the meaning of '2T{HH}' both unsurprising and easy

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: As a separate issue, I notice that the new 'T{}' code doesn't respect multiplicities, e.g., as in 'H3T{HHL}'. Is that intentional/desirable? That could have been an oversight on my part. I don't see any immediate reason why we wouldn't

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: (whoops -- used unsigned struct codes to correspond to signed C types there; but it shouldn't make any difference to the sizes and padding). I am tempted to suggest that we just go back to having one specifier at the beginning of the

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Granted, yes. But I wouldn't expect the same padding for 'BT{BI}' and 'BBI'. 'BT{BI}' should match a C struct which itself has an embedded struct. For C, I get the following results on my machine: I wasn't sure. The C99 standard does not

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: The C99 standard does not specify what the behavior should be. Right; it's down to the platform ABI. I think the least common multiple of the alignment requirements of the struct members is the way to go, though. It's difficult to

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Another snippet, from the latest public draft of the System V x86-64 ABI [1]: Structures and unions assume the alignment of their most strictly aligned compo- nent. Each member is assigned to the lowest available offset with the appropriate

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: One more reference: http://msdn.microsoft.com/en-us/library/9dbwhz68(v=VS.80).aspx gives essentially the same rules for MSVC. The alignment of the beginning of a structure or a union is the maximum alignment of any individual member. Each

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Thanks for the Rietveld upload. I haven't had a chance to review this properly yet, but hope to do so within the next few days. One question: the production list you added to the docs says: format_string: (`byte_order_specifier`?

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Travis, this issue is still assigned to you. Do you plan to work on this at some stage, or may I unassign you? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Hmm. Something's not quite right: the _struct module fails to compile for me with this patch. I get: /Users/dickinsm/python/svn/py3k/Modules/_struct.c: In function ‘s_unpack’: /Users/dickinsm/python/svn/py3k/Modules/_struct.c:1730: error:

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Ah, it should have been: assert(soself-s_tree != NULL); Got it now. All tests pass. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: One more design question. For something like: 'HT{H}H', what endianness should be used when packing/unpacking the last 'H'? Should the switch to '' within the embedded struct be regarded as local to the struct? With your patch, I get:

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Thanks for this. Any chance you could upload the patch to Rietveld (http://codereview.appspot.com/) for ease of review? -- ___ Python tracker rep...@bugs.python.org

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Sure - http://codereview.appspot.com/1258041 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Attached is a patch that implements part of the additions. More specifically, the 'T{}' syntax and the ability to place byte-order specifiers ('', '', '@', '^', '!, '=') anywhere in the struct string. The changes dictated by the PEP are so

[issue3132] implement PEP 3118 struct changes

Changes by Noufal nou...@nibrahim.net.in: -- nosy: +noufal ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list mailing

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: [Meador Inge] So the next step is to kick off a thread on python-dev summarizing the questions\problems we have come up with? I can get that started. Sounds good. I'd really like to see some examples of how these struct-module additions

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: About long doubles again: I just encountered someone on the #python IRC channel who wanted to know whether struct.pack and struct.unpack supported reading and writing of x87 80-bit long doubles (padded to 12 bytes each in the input). A

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: The main thing that I realized from this is that unpacking as a ctypes long double isn't all that useful for someone who wants to be able to do arithmetic on the unpacked result. I agree. Especially since ctypes 'long double' maps to a

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Mark, I think a lot of this discussion needs to go back to python-dev; with luck, we can get some advice and clarifications from the PEP authors there. So the next step is to kick off a thread on python-dev summarizing the

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: And under what conditions would a ctype long double be used vs. a Decimal object. Well, I'm guessing that this was really just an open question for the PEP, and that the PEP authors hadn't decided which of these two options was more

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: I'm looking for previous discussions of this PEP. There's a python-dev thread in April 2007: http://mail.python.org/pipermail/python-dev/2007-April/072537.html Are there other discussions that I'm missing? --

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Hi All, On Sat, Feb 13, 2010 at 5:07 AM, Mark Dickinson rep...@bugs.python.orgwrote: Mark Dickinson dicki...@gmail.com added the comment: Some of the proposed struct module additions look far from straightforward; I find that section of

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Some of the proposed struct module additions look far from straightforward; I find that section of the PEP significantly lacking in details and motivation. Unpacking a long-double will return a decimal object or a ctypes long-double.

[issue3132] implement PEP 3118 struct changes

Mark Dickinson dicki...@gmail.com added the comment: Whoops. ctypes does have long double, of course. Apologies. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___

[issue3132] implement PEP 3118 struct changes

Meador Inge mead...@gmail.com added the comment: Is anyone working on implementing these new struct modifiers? If not, then I would love to take a shot at it. -- nosy: +minge ___ Python tracker rep...@bugs.python.org

[issue3132] implement PEP 3118 struct changes

Benjamin Peterson benja...@python.org added the comment: 2010/2/12 Meador Inge rep...@bugs.python.org: Meador Inge mead...@gmail.com added the comment: Is anyone working on implementing these new struct modifiers?  If not, then I would love to take a shot at it. Not to my knowledge.

[issue3132] implement PEP 3118 struct changes

Travis Oliphant oliph...@enthought.com added the comment: On Feb 12, 2010, at 7:29 PM, Meador Inge wrote: Meador Inge mead...@gmail.com added the comment: Is anyone working on implementing these new struct modifiers? If not, then I would love to take a shot at it. That would be great.

[issue3132] implement PEP 3118 struct changes

Changes by Mark Dickinson dicki...@gmail.com: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3132 ___ ___ Python-bugs-list

[issue3132] implement PEP 3118 struct changes

Daniel Diniz aja...@gmail.com added the comment: Travis, Do you think you can contribute for this to actually land in 3.2? Having a critical issue slipping from 3.0 to 3.3 would be bad... Does this supersede issue 2395 or is this a subset of that one.? -- nosy: +ajaksu2 stage: - test

[issue3132] implement PEP 3118 struct changes

Travis Oliphant [EMAIL PROTECTED] added the comment: This can be re-targeted to 3.1 as described. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3132 ___ ___

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou [EMAIL PROTECTED] added the comment: Let's retarget it to 3.1 then. It's a new feature, not a behaviour change or a deprecation, so adding it to 3.0 isn't a necessity. -- components: +Library (Lib) nosy: +pitrou priority: release blocker - critical versions: +Python 3.1

[issue3132] implement PEP 3118 struct changes

Benjamin Peterson [EMAIL PROTECTED] added the comment: Actually, this may be a requirement of #2394; PEP 3118 states that memoryview.tolist would use the struct module to do the unpacking. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou [EMAIL PROTECTED] added the comment: Actually, this may be a requirement of #2394; PEP 3118 states that memoryview.tolist would use the struct module to do the unpacking. :-( However, we don't have any examples of the buffer API / memoryview object working with something else

[issue3132] implement PEP 3118 struct changes

Barry A. Warsaw [EMAIL PROTECTED] added the comment: It's looking pessimistic that this is going to make it by beta 3. If they can't get in by then, it's too late. -- nosy: +barry ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3132

[issue3132] implement PEP 3118 struct changes

Changes by Benjamin Peterson [EMAIL PROTECTED]: -- priority: critical - release blocker ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3132 ___ ___

[issue3132] implement PEP 3118 struct changes

Jean Brouwers [EMAIL PROTECTED] added the comment: If the struct changes are made, add also 2 formats for C types ssize_t and size_t, perhaps 'z' resp. 'Z'. In particular since on platforms sizeof(size_t) != sizeof(long). -- nosy: +MrJean1 ___

[issue3132] implement PEP 3118 struct changes

New submission from Benjamin Peterson [EMAIL PROTECTED]: It seems the new modifiers to the struct.unpack/pack module that were proposed in PEP 3118 haven't been implemented yet. -- assignee: teoliphant messages: 68347 nosy: benjamin.peterson, teoliphant priority: critical severity: