Re: [Python-ideas] Ideas for improving the struct module

On 21 January 2017 at 14:51, Nathaniel Smith wrote: > On Fri, Jan 20, 2017 at 7:39 PM, Nathaniel Smith wrote: >> [...] >> Some of these strategies that you might find helpful (or not): > > Oh right, and of course just after I hit send I realized I forgot one > of

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 7:39 PM, Nathaniel Smith wrote: > [...] > Some of these strategies that you might find helpful (or not): Oh right, and of course just after I hit send I realized I forgot one of my favorites! - come up with a real chunk of code from a real project that

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 3:37 PM, Elizabeth Myers wrote: [...] >> Some of the responses on the bug are discouraging... mostly seems to >> boil down to people just not wanting to expand the struct module or >> discourage its use. Everyone is a critic. I didn't know adding

Re: [Python-ideas] Ideas for improving the struct module

On 20/01/17 17:26, Elizabeth Myers wrote: > On 20/01/17 16:46, Cameron Simpson wrote: >> On 20Jan2017 14:47, Elizabeth Myers wrote: >>> 1) struct.unpack and struct.unpack_from should remain >>> backwards-compatible. I don't want to return extra values from it like >>>

Re: [Python-ideas] Ideas for improving the struct module

On 20/01/17 16:46, Cameron Simpson wrote: > On 20Jan2017 14:47, Elizabeth Myers wrote: >> 1) struct.unpack and struct.unpack_from should remain >> backwards-compatible. I don't want to return extra values from it like >> (length unpacked, (data...)) for that reason. > >

Re: [Python-ideas] Ideas for improving the struct module

On 20Jan2017 14:47, Elizabeth Myers wrote: 1) struct.unpack and struct.unpack_from should remain backwards-compatible. I don't want to return extra values from it like (length unpacked, (data...)) for that reason. Fully agree with this. If the calcsize solution

Re: [Python-ideas] Ideas for improving the struct module

On 20 January 2017 at 20:47, Elizabeth Myers wrote: > Two things: > > 1) struct.unpack and struct.unpack_from should remain > backwards-compatible. I don't want to return extra values from it like > (length unpacked, (data...)) for that reason. If the calcsize solution >

Re: [Python-ideas] Ideas for improving the struct module

On 20 January 2017 at 18:18, Guido van Rossum wrote: > I'd be wary of making a grab-bag of small improvements, it encourages > bikeshedding. Agreed. Plus the bikeshedding and debating risks draining Elizabeth's motivation. Paul

Re: [Python-ideas] Ideas for improving the struct module

I'd be wary of making a grab-bag of small improvements, it encourages bikeshedding. --Guido (mobile) On Jan 20, 2017 10:16 AM, "Ethan Furman" wrote: > On 01/20/2017 10:09 AM, Joao S. O. Bueno wrote: > >> On 20 January 2017 at 16:51, Elizabeth Myers wrote: >> > > Should I

Re: [Python-ideas] Ideas for improving the struct module

On 20 January 2017 at 15:13, Nathaniel Smith wrote: > On Jan 20, 2017 09:00, "Paul Moore" wrote: > > On 20 January 2017 at 16:51, Elizabeth Myers > wrote: >> Should I write up a PEP about this? I am not sure if it's justified or >>

Re: [Python-ideas] Ideas for improving the struct module

On 19/01/17 20:54, Cameron Simpson wrote: > On 19Jan2017 16:04, Yury Selivanov wrote: >> This is a neat idea, but this will only work for parsing framed >> binary protocols. For example, if you protocol prefixes all packets >> with a length field, you can write an

Re: [Python-ideas] Ideas for improving the struct module

On 19/01/17 20:40, Cameron Simpson wrote: > On 19Jan2017 12:08, Elizabeth Myers wrote: >> I also didn't mention that when you are unpacking iteratively (e.g., you >> have multiple strings), the code becomes a bit more hairy: >> > test_bytes =

Re: [Python-ideas] Ideas for improving the struct module

On 19/01/17 15:04, Yury Selivanov wrote: > This is a neat idea, but this will only work for parsing framed > binary protocols. For example, if you protocol prefixes all packets > with a length field, you can write an efficient read buffer and > use your proposal to decode all of message's fields

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 12:30 AM, Steven D'Aprano wrote: > Does it require a PEP just to add one more > format code? (Maybe it will, if the format code requires a complete > re-write of the entire module.) Yes, I think a PEP would be useful in this case. The proposed change

Re: [Python-ideas] Ideas for improving the struct module

Nevertheless the C meaning *is* the etymology of the module name. :-) --Guido (mobile) On Jan 19, 2017 16:54, "Chris Angelico" wrote: > On Fri, Jan 20, 2017 at 11:38 AM, Steven D'Aprano > wrote: > > On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 11:38 AM, Steven D'Aprano wrote: > On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris Angelico wrote: > >> To be fair, the name "struct" implies a C-style structure, which >> _does_ have a fixed size, or at least fixed offsets for its members > > > Ah,

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 05:16:28AM +1100, Chris Angelico wrote: > To be fair, the name "struct" implies a C-style structure, which > _does_ have a fixed size, or at least fixed offsets for its members Ah, the old "everyone thinks in C terms" fallacy raises its ugly head agan :-) The name

Re: [Python-ideas] Ideas for improving the struct module

On Thu, Jan 19, 2017 at 08:31:03AM +, Mark Dickinson wrote: > On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: > > [...] struct already supports > > variable-width formats. > > Unfortunately, that's not really true: the Pascal strings it supports > are in some

Re: [Python-ideas] Ideas for improving the struct module

Construct has radical API changes and should remain apart. It feels to me like a straw-man to introduce a large library to the discussion as justification for it being too-specialized. This proposal to me seems much more modest: add another format character (or two) to the existing set of a dozen

Re: [Python-ideas] Ideas for improving the struct module

I haven't had a chance to use it myself yet, but I've heard good things about https://construct.readthedocs.io/en/latest/ It's certainly far more comprehensive than struct for this and other problems. As usual, there's some tension between adding stuff to the stdlib versus using more

Re: [Python-ideas] Ideas for improving the struct module

ctypes.Structure is *literally* the interface to the C struct that as Chris mentions has fixed offsets for all members. I don't think that should (can?) be altered. In file formats (beyond net protocols) the string size + variable length string motif comes up often and I am frequently

Re: [Python-ideas] Ideas for improving the struct module

I am for upgrading struct to these, if possible. But besides my +1, I am writting in to remember folks thatthere is another "struct" model in the stdlib: ctypes.Structure - For reading a lot of records with the same structure it is much more handy than struct, since it gives one a suitable

Re: [Python-ideas] Ideas for improving the struct module

On Fri, Jan 20, 2017 at 5:08 AM, Elizabeth Myers wrote: > I do understand it might require a possible major rewrite or major > changes the struct module, but in the long run, I think it's worth it > (especially because the struct module is not all that big in scope). As

Re: [Python-ideas] Ideas for improving the struct module

On 19/01/17 05:58, Rhodri James wrote: > On 19/01/17 08:31, Mark Dickinson wrote: >> On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano >> wrote: >>> [...] struct already supports >>> variable-width formats. >> >> Unfortunately, that's not really true: the Pascal strings it

Re: [Python-ideas] Ideas for improving the struct module

On Thu, Jan 19, 2017 at 1:27 AM, Steven D'Aprano wrote: > [...] struct already supports > variable-width formats. Unfortunately, that's not really true: the Pascal strings it supports are in some sense variable length, but are stored in a fixed-width field. The internals of

Re: [Python-ideas] Ideas for improving the struct module

On Wed, Jan 18, 2017 at 04:24:39AM -0600, Elizabeth Myers wrote: > Hello, > > I've noticed a lot of binary protocols require variable length > bytestrings (with or without a null terminator), but it is not easy to > unpack these in Python without first reading the desired length, or > reading

Re: [Python-ideas] Ideas for improving the struct module

+1 on the idea of supporting variable-length strings with the length encoded in the preceding packed element! Several months ago I was trying to write a parser and writer of PostgreSQL's COPY ... WITH BINARY format. I started out trying to implement it in pure python using the struct module. Due