[issue3132] implement PEP 3118 struct changes

[Stefan Krah]

Changes by Stefan Krah :


--
versions: +Python 3.6 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Stefan Krah]

Stefan Krah added the comment:

Here's a grammar that roughly describes the subset that NumPy supports.

As for implementing this in the struct module: There is a new data
description language on the horizon:

  http://datashape.readthedocs.org/en/latest/


It does not have all the low-level capabilities (e.g changing alignment
on the fly), but it is far more readable. Example:

PEP-3118:  "(2,3)10f0fZdT{10B:x:(2,3)d:y:Q:z:}B"
Datashape: "2 * 3 * (10 * float32, 0 * float32, complex128, {x: 10 * uint8, y: 
2 * 3 * float64, z: int64}, uint8)"


There are a lot of open questions still. Should "10f" be viewed as an
array[10] of float, i.e. equivalent to (10)f?

In the context of PEP-3118, I think so.

--
nosy: +skrah
Added file: http://bugs.python.org/file42451/grammar.y

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Stefan Krah]

Changes by Stefan Krah stefan-use...@bytereef.org:


--
nosy:  -skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Paul Hoffman]

Paul Hoffman added the comment:

Following up on 
http://mail.python.org/pipermail/python-ideas/2011-March/009656.html, I would 
like to request that struct also handle half-precision floats directly. It's a 
short change, and half-precision floats are becoming much more popular in 
applications.

Adding this to struct would also maybe need to change math.isinf and 
math.isnan, but maybe not.

--
nosy: +paulehoffman

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson added the comment:

Paul: there's already an open issue for adding float16 to the struct module: 
see issue 11734.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Paul Hoffman]

Paul Hoffman added the comment:

Whoops, never mind. Thanks for the pointer to 11734.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Alexander Belopolsky]

Changes by Alexander Belopolsky alexander.belopol...@gmail.com:


--
nosy: +belopolsky -Alexander.Belopolsky

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Nick Coghlan]

Nick Coghlan added the comment:

Following up here after rejecting #15622 as invalid

The unicode codes in PEP 3118 need to be seriously rethought before any 
related changes are made in the struct module.

1. The 'c' and 's' codes are currently used for raw bytes data (represented as 
bytes objects at the Python layer). This means the 'c' code cannot be used as 
described in PEP 3118 in a world with strict binary/text separation.

2. Any format codes for UCS1, UCS2 and UCS4 are more usefully modelled on 's' 
than they are on 'c' (so that repeat counts create longer strings rather than 
lists of strings that each contain a single code point)

3. Given some of the other proposals in PEP 3118, it seems more useful to 
define an embedded text format as S{encoding}.

UCS1 would then be S{latin-1}, UCS2 would be approximated as S{utf-16} and 
UCS4 would be S{utf-32} and arbitrary encodings would also be supported. 
struct packing would implicitly encode from text to bytes while unpacking would 
implicitly decode bytes to text. As with 's' a length mismatch in the encoded 
form would mean an error.

--
nosy: +ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Stefan Krah]

Stefan Krah stefan-use...@bytereef.org added the comment:

Yes, definitely. I'm going to push a new memoryview implementation
(complete for all 1D/native format cases) in a couple of days.

Once that is done, perhaps we could create a memoryview-struct
branch on top of that.

--
nosy: +skrah

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Is this work something that might be suitable for the features/pep-3118 repo 
(http://hg.python.org/features/pep-3118/) ?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Is there still any interest in this work?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Yes, there's interest (at least here).  I've just been really short on 
Python-time recently, so haven't found time to review your patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

I'm going to unassign for now;  I still hope to look at this at some point, but 
can't see a time in the near future when it's going to happen.

--
assignee: mark.dickinson - 

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
assignee: meador.inge - mark.dickinson
stage: needs patch - patch review
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Attached is the latest version of the struct string patch.  I tested on OS X 
10.6.5 (64-bit) and Ubuntu 10.04 (32-bit).  I also scanned for memory problems 
with Valgrind.  There is one test failing on 32-bit systems ('test_crasher').  
This is due to the fact that 'struct.pack(357913941b, ...)' no longer tries 
to allocate 357913941 format codes.  This implementation just allocates *one* 
code and assigns a count of 357913941, which is utilized later when 
packing/unpacking.  Some work could be done to add better large memory 
consumption checks, though.

Previous feedback has been incorporated:

   1. Multiplicities allowed on struct specifiers.
   2. Maximum alignment rule.
   3. Struct nesting depth limited (64 levels).
   4. The old behavior of only one byte order specified.  However, 
  the code is written in a way such that the scoped behavior
  would be easy to add.

As before, there will surely be more iterations, but this is good enough for 
general review to see if things are headed in the right direction.  

This is a difficult one for review because the diffs are really large.  I 
placed a review on Rietveld here: http://codereview.appspot.com/3863042/.  If 
anyone has any ideas on how to reduce the number number of diffs (perhaps a way 
to do multiple smaller pataches), then that would be cool.  I don't see an 
obvious way to do this at this point.

--
Added file: http://bugs.python.org/file20298/struct-string.py3k.3.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 Another possibility is to implement the 'O' format unsafely [...]

Hmm.  I don't much like that idea.  Historically, it's supposed to be very 
difficult to segfault the Python interpreter with pure Python code (well except 
if you're using ctypes, I guess).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 For reference, Numpy's PEP 3118 implementation is here:

Thanks for that, and the other information you give;  that's helpful.

It sounds like we're on the same page with respect to alignment of substructs.  
(Bar the mostly academic question of max versus lcm.)

I still like the idea of scoped endianness markers in the substructs, but  if 
we have to abandon that for compatibility with NumPy that's okay.

- I assumed the 'O' format in the PEP is supposed to be similar to Numpy
  object arrays. This implies some reference counting semantics. The
  Numpy PEP 3118 implementation assumes the memory contains borrowed
  references, valid at least until the buffer is released.
  Unpacking 'O' should probably INCREF whatever PyObject* pointer is
  there.

I'm still confused about how this could work:  when unpacking, how do you know 
whether the PyObject* pointer points to a valid object or not?  You can ensure 
that the pointer will always point to a valid object by having the *pack* 
operation increment reference counts, but then you need a way to automatically 
decref when the packed string goes out of scope.  So the object returned by 
'pack' would somehow have to be something other than a plain string, so that it 
can deal with automatically doing the DECREF of the held PyObject* pointers 
when it goes out of scope.

What's the need to have the 'O' format in the struct module?  Is it really 
necessary there?  Can we get away with not implementing it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 For reference, Numpy's PEP 3118 implementation is here

BTW, does this already exist in a released version of NumPy?  If not, when is 
it likely to appear in the wild?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Pauli Virtanen]

Pauli Virtanen p...@iki.fi added the comment:

 I still like the idea of scoped endianness markers in the substructs,
 but  if we have to abandon that for compatibility with NumPy that's
 okay.

That, or change the Numpy implementation. I don't believe there's yet much code 
in the wild that changes the alignment specifier on the fly.

[clip: 'O' format code]
 So the object returned by 'pack' would somehow
 have to be something other than a plain string, so that it can deal
 with automatically doing the DECREF of the held PyObject* pointers
 when it goes out of scope.

Yes, the packed object would need to own the references, and it would be the 
responsibility of the provider of the buffer to ensure that the pointers are 
valid.

It seems that it's not possible for the `struct` module to correctly implement 
packing for the 'O' format. Unpacking could be possible, though (but then if 
you don't have packing, how write tests for it?).

Another possibility is to implement the 'O' format unsafely and leave managing 
the reference counting to whoever uses the `struct` module's capabilities. (And 
maybe return ctypes pointers on unpacking.)

[clip]
 What's the need to have the 'O' format in the struct module?  Is it
 really necessary there?  Can we get away with not implementing it?

Numpy arrays, when containing Python objects, function as per the 'O' format.

However, for the struct module, I don't see what would be the use case for the 
'O' format.

 BTW, does this already exist in a released version of NumPy?  If not,
 when is it likely to appear in the wild?

It's included since the 1.5.0 release which came out last July.

***

I think after the implementation is done, the PEP probably needs to be amended 
with clarifications (and possibly cutting out what is not really needed).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Pauli Virtanen]

Pauli Virtanen p...@iki.fi added the comment:

For reference, Numpy's PEP 3118 implementation is here:

http://github.com/numpy/numpy/blob/master/numpy/core/_internal.py#L357

http://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/buffer.c#L76

It would be a good idea to ensure that the numpy and struct implementations are 
in agreement about details of the format strings.
(I wouldn't take the Numpy implementation as the definitive one, though.)

- The sub-structs in Numpy arrays (in align=True mode) are aligned
  according to the maximum alignment of the fields.

- I assumed the 'O' format in the PEP is supposed to be similar to Numpy
  object arrays. This implies some reference counting semantics. The
  Numpy PEP 3118 implementation assumes the memory contains borrowed
  references, valid at least until the buffer is released.
  Unpacking 'O' should probably INCREF whatever PyObject* pointer is
  there.

- I assumed the alignment specifiers were unscoped. I'm not sure
  however whether this is the best thing to do.

- The function pointers and pointers to pointers were not implemented.
  (Numpy cannot represent those as data types.)

--
nosy: +pv

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Changes by Meador Inge mead...@gmail.com:


--
assignee: teoliphant - meador.inge
priority: critical - high
stage: unit test needed - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Travis Oliphant]

Travis Oliphant oliph...@enthought.com added the comment:

On May 19, 2010, at 2:38 PM, Mark Dickinson wrote:

 
 Mark Dickinson dicki...@gmail.com added the comment:
 
 Travis, this issue is still assigned to you.  Do you plan to work on this at 
 some stage, or may I unassign you?
 

You may unassign it from me.   Unfortunately, I don't have time anymore to work 
on it and I don't see that changing in the coming months. 

Thanks,

-Travis

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

 is that correct, or should the production list be something like:

Yup, you are right.  I will change the grammar.

 Whether these cases are valid or not (personally, I think they should 
 be), we should add some tests for them.  '' *is* currently valid, I 
 believe.

I agree, they should be valid.  I will add more test cases.

 The possibility of mixing native size/alignment with standard 
 size/alignment in a single format string makes me a bit uneasy

I agree.  It is hard for me to see how this might be used.  In any case,
the relevant part of the PEP that I was following is:

Endian-specification ('!', '@','=','','', '^') is also allowed inside the 
string so that it can change if needed. The previously-specified endian string 
is in force until changed. The default endian is '@' which means native 
data-types and alignment. If un-aligned, native data-types are requested, then 
the endian specification is '^'.

However, I am not quite sure how to interpret the last sentence.

 Should the switch to '' within the embedded struct be regarded as 
 local to the struct?

No, there is no notion of scope here.  A given specifier is active until the 
next one is found.

 Ah, it should have been:
 
 assert(soself-s_tree != NULL);

D'oh!  I missed that when I merge over to py3k -- I started this work on trunk. 
 Thanks.

--
Added file: http://bugs.python.org/file17416/struct-string.py3k.2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  The possibility of mixing native size/alignment with standard 
  size/alignment in a single format string makes me a bit uneasy
 
 I agree.  It is hard for me to see how this might be used.

Without having anything more constructive to add, I also agree with this
gut feeling. Perhaps not all of the PEP needs implementing; we can just
add what is genuinely useful.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Thanks for the new patch.

 ... If un-aligned, native data-types are requested, then the
 endian specification is '^'.

 However, I am not quite sure how to interpret the last sentence.

Hmm.  Seems like the PEP authors are proposing a new byteorder/alignment/size 
specifier here:  '^' = native byte-order + native size + no alignment.  I 
missed this before.

 Should the switch to '' within the embedded struct be regarded as 
 local to the struct?

No, there is no notion of scope here.  A given specifier is active until the 
next one is found.

Okay.  I wonder whether that's the most useful thing to do, though.

As a separate issue, I notice that the new 'T{}' code doesn't respect 
multiplicities, e.g., as in 'H3T{HHL}'.  Is that intentional/desirable?

 struct.pack('H3T{HHL}', 1, (2, 3, 4))
b'\x01\x00\x02\x00\x03\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00'

If we don't allow multiplicities, this should produce an exception, I think.  
If we do allow multiplicities (and I don't immediately see why we shouldn't), 
then we're going to have to be clear about how endianness behaves in something 
like:

'H3T{HH}'

So the first inner struct here would be treated as '{HH}'.  Would the next 
two be identical to this, or would they be as though the whole thing were 
'HT{HH}T{HH}T{HH}', in which case the 2nd and 3rd substructs are both 
effectively 'HH', while the first is 'HH'.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

After a bit more thought (and after soliciting a couple of opinions on 
#python-dev), I'm convinced that endianness changes within a substruct should 
be local to that substruct:

- it makes the meaning of '2T{HH}' both unsurprising and easy to understand:  
i.e., it would be interpreted exactly as 'T{HH}T{HH}', and  both substructs 
would behave like 'HH'.

- I suspect it's the behaviour that people expect

- it may make dynamic creation of struct format strings easier/less bug-prone.


But now I've got a new open issue:  how much padding should be 
inserted/expected (for pack/unpack respectively) between the 'B' and the 
'T{...}' in a struct format string of the form 'BT{...}'?

For this, I don't think we can hope to exactly comply with the platform ABI in 
all cases.  But I think there's a simple rule that would work 99% of the time, 
and that does match the ABI on common platforms (though I'll check this), 
namely that the alignment requirement of the 'T{...}' struct should be the 
least common multiple of the alignment requirements of any of its elements.  
(Which will usually translate to the largest of the alignment requirements, 
since those alignments are usually all going to be powers of 2.)

And *this* is where things get tricky if the alignment/byteorder/size specifier 
is changed midstream, since then it doesn't seem clear what alignments would 
contribute to the lcm above.  I'm tempted to suggest that for native mode, 
changing the specifier be disallowed entirely.

Travis, any comments on any of this?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

 As a separate issue, I notice that the new 'T{}' code doesn't respect 
 multiplicities, e.g., as in 'H3T{HHL}'.  Is that 
 intentional/desirable?

That could have been an oversight on my part.  I don't see any immediate reason 
why we wouldn't allow it.

 But now I've got a new open issue:  how much padding should be 
 inserted/expected (for pack/unpack respectively) between the 'B' and 
 the 'T{...}' in a struct format string of the form 'BT{...}'?

Doesn't that depend on what is in the '...'?  For example, I would expect the 
same padding for 'BT{I}' and 'BI'.  In general, I would expect the padding to 
be the same for 'x+T{y+}' and 'x+y+'.  The 'T{...}'s are merely organizational, 
right?

 I'm tempted to suggest that for native mode, changing the specifier be 
 disallowed entirely.

I am tempted to suggest that we just go back to having one specifier at the 
beginning of the string :).  Things seem to be getting complicate without any 
clear benefits.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

(whoops -- used unsigned struct codes to correspond to signed C types there;  
but it shouldn't make any difference to the sizes and padding).

 I am tempted to suggest that we just go back to having one specifier at the 
 beginning of the string :).  Things seem to be getting complicate without any 
 clear benefits.

Agreed.  Though if anyone following this issue wants to make the case that 
there are benefits to being able to change the endianness midway through a 
string, please do so!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

 Granted, yes.  But I wouldn't expect the same padding for 'BT{BI}' and 
 'BBI'.  'BT{BI}' should match a C struct which itself has an embedded 
 struct.  For C, I get the following results on my machine:

I wasn't sure.  The C99 standard does not specify what the behavior should be.  
It is implementation defined.  I guess most implementations just set the 
alignment of the struct with the alignment of its most demanding member.

I need to change how the alignment for nested structures is computed.  Right 
now alignments are being computed as if the 'T{...}' codes were not there.  I 
will hold off until we decide what that rule should be, but I think the most 
demanding element rule seems reasonable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 The C99 standard does not specify what the behavior should be.

Right;  it's down to the platform ABI.

I think the least common multiple of the alignment requirements of the struct 
members is the way to go, though.  It's difficult to imagine an ABI for which 
this lcm isn't the same thing as the largest struct member alignment, but I 
don't want to categorically say that such ABIs don't exist.

Here's a snippet from the gcc manual [1]:

Note that the alignment of any given struct or union type is required by the 
ISO C standard to be at least a perfect multiple of the lowest common multiple 
of the alignments of all of the members of the struct or union in question.

I'm not sure I could identify the precise pieces of the standard that imply 
that requirement, though.

[1] 
http://gcc.gnu.org/onlinedocs/gcc-4.5.0/gcc/Type-Attributes.html#Type-Attributes

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Another snippet, from the latest public draft of the System V x86-64 ABI [1]:

Structures and unions assume the alignment of their most strictly aligned 
compo- nent. Each member is assigned to the lowest available offset with the 
appropriate alignment. The size of any object is always a multiple of the 
object‘s alignment.

I'd be fine with using the largest alignment, as above, instead of computing an 
lcm;  I can't believe it'll ever make a difference in practice.  For an empty 
struct (not allowed in C99, but allowed as a gcc extension, and allowed by the 
struct module), the alignment would be 1, of course.

[1] http://www.x86-64.org/documentation/abi.pdf

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

One more reference:

http://msdn.microsoft.com/en-us/library/9dbwhz68(v=VS.80).aspx

gives essentially the same rules for MSVC. The alignment of the beginning of a 
structure or a union is the maximum alignment of any individual member. Each 
member within the structure or union must be placed at its proper alignment as 
defined in the previous table, which may require implicit internal padding, 
depending on the previous member.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Thanks for the Rietveld upload.  I haven't had a chance to review this properly 
yet, but hope to do so within the next few days.

One question: the production list you added to the docs says:

  format_string: (`byte_order_specifier`? `type_string`)*

This suggests that format strings like '' and 'b' are invalid;  is that 
correct, or should the production list be something like:

  format_string: (`byte_order_specifier` | `type_string`)*

?  Whether these cases are valid or not (personally, I think they should be), 
we should add some tests for them.  '' *is* currently valid, I believe.


The possibility of mixing native size/alignment with standard size/alignment in 
a single format string makes me a bit uneasy, but I can't see any actual 
problems that might arise from it (equally, I can't imagine why anyone would 
want to do it).  I wondered briefly whether padding has clear semantics when a 
'@' appears in the middle of a format string, but I can't see why it wouldn't 
have.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Travis, this issue is still assigned to you.  Do you plan to work on this at 
some stage, or may I unassign you?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Hmm.  Something's not quite right: the _struct module fails to compile for me 
with this patch.  I get:

/Users/dickinsm/python/svn/py3k/Modules/_struct.c: In function ‘s_unpack’:
/Users/dickinsm/python/svn/py3k/Modules/_struct.c:1730: error: ‘PyStructObject’ 
has no member named ‘s_codes’
/Users/dickinsm/python/svn/py3k/Modules/_struct.c: In function ‘s_unpack_from’:
/Users/dickinsm/python/svn/py3k/Modules/_struct.c:1765: error: ‘PyStructObject’ 
has no member named ‘s_codes’

The offending lines both look like:

assert(soself-s_codes != NULL);

presumably that should be:

assert(soself-s_tree-s_codes != NULL);

After making that change, and successfully rebuilding, this assert triggers:

test_705836 (__main__.StructTest) ... Assertion failed: 
(soself-s_tree-s_codes != NULL), function s_unpack, file 
/Users/dickinsm/python/svn/py3k/Modules/_struct.c, line 1730.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Ah, it should have been:

assert(soself-s_tree != NULL);

Got it now.  All tests pass. :)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

One more design question.  For something like: 'HT{H}H', what endianness 
should be used when packing/unpacking the last 'H'?  Should the switch to '' 
within the embedded struct be regarded as local to the struct?

With your patch, I get:

 pack('HT{H}H', 1, (2,), 3)
b'\x01\x00\x00\x02\x00\x03'

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Thanks for this.

Any chance you could upload the patch to Rietveld 
(http://codereview.appspot.com/) for ease of review?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Sure - http://codereview.appspot.com/1258041

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Attached is a patch that implements part of the additions.  More specifically, 
the 'T{}' syntax and the ability to place byte-order specifiers ('', '', '@', 
'^', '!, '=') anywhere in the struct string.  

The changes dictated by the PEP are so big that it is better to split things up 
into multiple patches.  These two features will lay some ground work and are 
probably less controversial than the others.

Surely some more tweaks will be needed, but I think what I have now is at least 
good enough for review.  I tested on OS X 10.6 and Ubuntu 10.4.  I also used 
valgrind and 'regrtest.py -R:' to check for memory and 
reference leaks, respectively.

--
Added file: http://bugs.python.org/file17386/struct-string.py3k.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Noufal]

Changes by Noufal nou...@nibrahim.net.in:


--
nosy: +noufal

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

[Meador Inge]
 So the next step is to kick off a thread on python-dev summarizing the
 questions\problems we have come up with?  I can get that started.

Sounds good.  I'd really like to see some examples of how these struct-module 
additions would be used in real life.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

About long doubles again:  I just encountered someone on the #python IRC 
channel who wanted to know whether struct.pack and struct.unpack supported 
reading and writing of x87 80-bit long doubles (padded to 12 bytes each in the 
input).  A few quotes from him/her, with permission (responses from others, 
including me, edited out;  I can supply a fuller transcript if necessary, but I 
hope what's below isn't misleading).

[18:39] bdesk: Hi, is struct.pack able to handle 80-bit x86 extended floats?
[18:40] bdesk: How can I read and write these 80-bit floats, in binary, using 
python?
[18:44] bdesk: dickinsm: I have a C program that uses binary files as input and 
output, and I want to deal with these files using python if possible.
[18:49] bdesk: I don't need to do arithmetic with the full 80 bits of precision 
within the python program, although It would be better if I could.
[18:50] bdesk: I would need to use the float in a more semantically useful 
manner than treating it as a black box of 12 bytes.
[18:55] bdesk: Until python gets higher precision floats, my preferred 
interface would be to lose some precision when unpacking the floats.

The main thing that I realized from this is that unpacking as a ctypes long 
double isn't all that useful for someone who wants to be able to do arithmetic 
on the unpacked result.  And if you don't want to do arithmetic on the unpacked 
result, then you're probably just shuffling the bytes around without caring 
about their meaning, so there's no need to unpack as anything other than a 
sequence of 12 bytes.

On the other hand, I suppose it's enough to be able to unpack as a ctypes 
c_longdouble and then convert to a Python float (losing precision) for the 
arithmetic.  Alternatively, we might consider simply unpacking a long double 
directly into a Python float (and accepting the loss of precision);  that seems 
to be what would be most useful for the use-case above.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

 The main thing that I realized from this is that unpacking as a ctypes long 
 double isn't all that useful for someone who wants to be able to do arithmetic
 on the unpacked result.  

I agree.  Especially since ctypes 'long double' maps to a Python float and
'.value' would have to be referenced on the ctype 'long double' instance
for doing arithmetic.

 And if you don't want to do arithmetic on the unpacked result, then you're 
 probably just shuffling the bytes around without caring about their meaning,
 so there's no need to unpack as anything other than a sequence of 12 bytes.

One benefit of having a type code for 'long double' (assuming you are mapping
the value to the platform's 'long double') is that the you don't have to know 
how many bytes are in the underlying representation.  As you know, it isn't 
always just 12 bytes.  It depends on the architecture and ABI being used.  Which
from a quick sample, I am seeing can be anywhere from 8 to 16
bytes:

===
| Compiler  | Arch | Bytes|
===
| VC++ 8.0  | x86  | 8|
| VC++ 9.0  | x86  | 8|
| GCC 4.2.4 | x86  | 12 (default), 16 |
| GCC 4.2.4 | x86-64   | 12, 16 (default) |  
| GCC 4.2.4 | PPC IBM  | 16   |
| GCC 4.2.4 | PPC IEEE | 16   |
===

 On the other hand, I suppose it's enough to be able to unpack as a ctypes 
 c_longdouble and then convert to a Python float (losing precision) for the 
 arithmetic.  Alternatively, we might consider simply unpacking a long double 
 directly into a Python float (and accepting the loss of precision);
 
I guess that would be acceptable.  The only thing that I don't like is that
since the transformation is lossy, you can't round trip:

   # this will not hold
   pack('g', unpack('g', byte_str)[0]) == byte_str

 that seems to be what would be most useful for the use-case above.

Which use case?  From the given IRC trace it seems that 'bdesk' was mainly
concerned with (1) pushing bytes around, but (2) thought it it would be better
to be able to do arithmetic and thought it would be more useful if it were
not a black box of 12 bytes.  For use case (1) the loss of precision would 
probably not be acceptable, due to the round trip issue mentioned above.

So using ctypes 'long double' is easier to implement, but is lossy and clunky 
for arithmetic.  Using Python 'float' is easy to implement and easy for 
arithmetic, but is lossy.  Using Decimal is non-lossy and easy for arithmetic, 
but the implementation would be non-trivial and architecture specific 
(unless we just picked a fixed number of bytes regardless of the architecture).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Mark,

 I think a lot of this discussion needs to go back to python-dev;  with 
 luck, we can get some advice and clarifications from the PEP authors 
 there.

So the next step is to kick off a thread on python-dev summarizing the 
questions\problems we have come up with?  I can get that started.

 Are there other discussions that I'm missing?

I did a quick search and came up with the same.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

 And under what conditions would a ctype long double be used vs. a
 Decimal object.

Well, I'm guessing that this was really just an open question for the PEP, and 
that the PEP authors hadn't decided which of these two options was more 
appropriate.  If all long doubles were converted to Decimal, then we need to 
determine what precision is appropriate to use for the conversion: any long 
double *can* be represented exactly as a Decimal, but to get an exact 
representation can need thousands of digits in some cases, so it's probably 
better to always round to some fixed number of signficant digits.  36 
significant digits is a reasonable choice here: it's the minimum number of 
digits that's guaranteed to distinguish two distinct long doubles, for the case 
where a long double has 113 bits of precision (i.e., IEEE 754 binary128 
format);  other common long double formats have smaller precision than this 
(usually 53 (normal double), 64 (x87 extended doubles), or 106 (double 
double)).  There would probably also need to be some way to 'repack' the 
Decimal instance.

The 'platform long double - Decimal' conversion itself would also be 
nontrivial to implement;  I can lend a hand here if you want it.

Using ctypes makes more sense to me, since it doesn't involve trying to mix 
decimal and binary, except that I don't know whether it's acceptable for other 
standard library modules to have dependencies on ctypes.  I'm not sure whether 
ctypes is available on all platforms that Python runs on. It's also a bit ugly 
that, depending on the platform, sometimes a long double will unpack to an 
instance of ctypes.long_double, and sometimes (when long double == double) to a 
regular Python float.

Anyway, this particular case (long double) isn't a big deal:  it can be 
overcome, one way or another.  I'm more worried about some of the other aspects 
of the changes.

[About unpacking with the 'O' format.]
 I guess if an object associated with the packed address does not
 exist, then you would unpack None (?).  This is especially a problem 
 if the struct-sting is being sent over the wire to another machine.

And how do you determine whether an address gives a valid object or not?  I can 
only assume that packing and unpacking with the 'O' format is only supposed to 
be used in certain restricted circumstances, but it's not clear to me what 
those circumstances are.

 I also have the following questions: [...]

I think a lot of this discussion needs to go back to python-dev;  with luck, we 
can get some advice and clarifications from the PEP authors there.  I'm not 
sure whether it's appropriate to modify the original PEP (especially since it's 
already accepted), or whether it would be better to produce a separate document 
describing the proposed changes in detail.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

I'm looking for previous discussions of this PEP.  There's a python-dev thread 
in April 2007:

http://mail.python.org/pipermail/python-dev/2007-April/072537.html

Are there other discussions that I'm missing?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Hi All,

On Sat, Feb 13, 2010 at 5:07 AM, Mark Dickinson rep...@bugs.python.orgwrote:


 Mark Dickinson dicki...@gmail.com added the comment:

 Some of the proposed struct module additions look far from straightforward;
  I find that section of the PEP significantly lacking in details and
 motivation.


I agree.

 Unpacking a long-double will return a decimal object or a ctypes
 long-double.

 Returning a Decimal object here doesn't make a lot of sense, since Decimal
 objects aren't generally compatible with floats.  And ctypes long double
 objects don't seem to exist, as far as I can tell.  It might be better not
 to add this code.

And under what conditions would a ctype long double be used vs. a Decimal
object.



Another bit that's not clear to me:  how is unpacking an object pointer
 expected to work, and how would it typically be used?  What if the unpacked
 pointer no longer points to a valid Python object?  How would this work in
 other Python implementations?


I guess if an object associated with the packed address does not exist, then
you would unpack None (?).  This is especially a problem if the struct-sting
is being sent over the wire to another machine.

 For the 'X{}' format (pointer to a function), is this supposed to mean a
 Python function or a C function?


I read that as a Python function.  However, I am not completely sure how the
prototype would be enforced when unpacking.  I am also wondering, though,
how the signatures on pointers-to-functions are specified?  Are
the arguments and return type full struct strings as well?

 What's a 'specific pointer'?

I think this means a pointer to a specific type, e.g. 'd' is a pointer to a
double. If this is the case, though, the use cases are not completely clear
to me.

I also have the following questions:

* Can pointers be nested, 'd' ?
* What nesting level can structures have? Arbitrary?
* The new array syntax claims multi-dimensional array of whatever follows.

  Truly whatever? Arrays of structures? Arrays of pointers?
* complex (whatever the next specifier is).  Not really 'whatever'.  You
  can not have a 'complex bool' or 'complex int'.  What other types of
  complex are there besides complex double?
* How do array specifiers and pointer specifiers mix?  For example, would
  '(2, 2)d' be a two-by-two array of pointers to doubles?  What about
  '(2, 2)d'?  Is this a pointer to an two-by-two array of doubles?

The new features of the struct-string syntax are so different that I think
we
need to specify a grammar.  I think it will clarify some of the open
questions.

In addition, I was thinking that a reasonable implemenation strategy would
be to keep the current struct-string syntax mostly in place within the C
module
implementation.  The C implementation would just provide an interface to
pack\unpack sequences of primitive data elements.  Then we could write a
layer in the Python 'struct' module that took care of the higher-order
concepts like nested structures, arrays, named values, and pointers to
functions.  The higher-order concepts would be mapped to the appropriate
primitive sequence strings.

I think this will simplify the implementation and will provide a way to
phase
it.  We can implement the primitive type extensions in C first followed by
the higher-level Python stuff.  The result of each phase is immediately
usuable.

I have attached a patch against the PEP containing my current thoughts on
fleshing out the grammar and some of the current open questions.  This still
needs work, but I wanted to share to see if I am on the right track.
 Please advise on how to proceed.

--
keywords: +patch
Added file: http://bugs.python.org/file16241/unnamed
Added file: http://bugs.python.org/file16242/pep-3118.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___divHi All,/divdivbr/divOn Sat, Feb 13, 2010 at 5:07 AM, Mark 
Dickinson span dir=ltrlt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span 
wrote:brdiv class=gmail_quoteblockquote class=gmail_quote 
style=margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;
br
Mark Dickinson lt;a 
href=mailto:dicki...@gmail.com;dicki...@gmail.com/agt; added the 
comment:br
br
Some of the proposed struct module additions look far from straightforward;  I 
find that section of the PEP significantly lacking in details and 
motivation.br/blockquotediv /divdivI 
agree./divdiv /divblockquote class=gmail_quote style=margin:0 0 0 
.8ex;border-left:1px #ccc solid;padding-left:1ex;

quot;Unpacking a long-double will return a decimal object or a ctypes 
long-double.quot;br
br
Returning a Decimal object here doesn#39;t make a lot of sense, since Decimal 
objects aren#39;t generally compatible with floats.  And ctypes long double 
objects don#39;t seem to exist, as far as I can tell.  It might be better not 
to add 

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Some of the proposed struct module additions look far from straightforward;  I 
find that section of the PEP significantly lacking in details and motivation.

Unpacking a long-double will return a decimal object or a ctypes long-double.

Returning a Decimal object here doesn't make a lot of sense, since Decimal 
objects aren't generally compatible with floats.  And ctypes long double 
objects don't seem to exist, as far as I can tell.  It might be better not to 
add this code.

Another bit that's not clear to me:  how is unpacking an object pointer 
expected to work, and how would it typically be used?  What if the unpacked 
pointer no longer points to a valid Python object?  How would this work in 
other Python implementations?

For the 'X{}' format (pointer to a function), is this supposed to mean a Python 
function or a C function?

What's a 'specific pointer'?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Mark Dickinson dicki...@gmail.com added the comment:

Whoops.  ctypes does have long double, of course.  Apologies.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Is anyone working on implementing these new struct modifiers?  If not, then I 
would love to take a shot at it.

--
nosy: +minge

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Benjamin Peterson]

Benjamin Peterson benja...@python.org added the comment:

2010/2/12 Meador Inge rep...@bugs.python.org:

 Meador Inge mead...@gmail.com added the comment:

 Is anyone working on implementing these new struct modifiers?  If not, then I 
 would love to take a shot at it.

Not to my knowledge.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Travis Oliphant]

Travis Oliphant oliph...@enthought.com added the comment:

On Feb 12, 2010, at 7:29 PM, Meador Inge wrote:


 Meador Inge mead...@gmail.com added the comment:

 Is anyone working on implementing these new struct modifiers?  If  
 not, then I would love to take a shot at it.

That would be great.

-Travis

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Mark Dickinson]

Changes by Mark Dickinson dicki...@gmail.com:


--
nosy: +mark.dickinson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Daniel Diniz]

Daniel Diniz aja...@gmail.com added the comment:

Travis,
Do you think you can contribute for this to actually land in 3.2? Having
a critical issue slipping from 3.0 to 3.3 would be bad...

Does this supersede issue 2395 or is this a subset of that one.?

--
nosy: +ajaksu2
stage:  - test needed
versions: +Python 3.2 -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Travis Oliphant]

Travis Oliphant [EMAIL PROTECTED] added the comment:

This can be re-targeted to 3.1 as described.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Antoine Pitrou]

Antoine Pitrou [EMAIL PROTECTED] added the comment:

Let's retarget it to 3.1 then. It's a new feature, not a behaviour
change or a deprecation, so adding it to 3.0 isn't a necessity.

--
components: +Library (Lib)
nosy: +pitrou
priority: release blocker - critical
versions: +Python 3.1 -Python 3.0

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Benjamin Peterson]

Benjamin Peterson [EMAIL PROTECTED] added the comment:

Actually, this may be a requirement of #2394; PEP 3118 states that
memoryview.tolist would use the struct module to do the unpacking.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Antoine Pitrou]

Antoine Pitrou [EMAIL PROTECTED] added the comment:

 Actually, this may be a requirement of #2394; PEP 3118 states that
 memoryview.tolist would use the struct module to do the unpacking.

:-(
However, we don't have any examples of the buffer API / memoryview
object working with something else than 1-dimensional contiguous char
arrays (e.g. bytearray). Therefore, I suggest that Python 3.0 provide
official support only for 1-dimensional contiguous char arrays. Then
tolist() will be easy to implement even without using the struct module
(just a list of integers, if I understand the functionality).

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Barry A. Warsaw]

Barry A. Warsaw [EMAIL PROTECTED] added the comment:

It's looking pessimistic that this is going to make it by beta 3.  If
they can't get in by then, it's too late.

--
nosy: +barry

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Benjamin Peterson]

Changes by Benjamin Peterson [EMAIL PROTECTED]:


--
priority: critical - release blocker

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Jean Brouwers]

Jean Brouwers [EMAIL PROTECTED] added the comment:

If the struct changes are made, add also 2 formats for C types ssize_t and 
size_t, perhaps 'z' resp. 'Z'.  In particular since on platforms 
sizeof(size_t) != sizeof(long).

--
nosy: +MrJean1

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3132] implement PEP 3118 struct changes

[Benjamin Peterson]

New submission from Benjamin Peterson [EMAIL PROTECTED]:

It seems the new modifiers to the struct.unpack/pack module that were
proposed in PEP 3118 haven't been implemented yet.

--
assignee: teoliphant
messages: 68347
nosy: benjamin.peterson, teoliphant
priority: critical
severity: normal
status: open
title: implement PEP 3118 struct changes
type: feature request
versions: Python 3.0

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3132
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com