Re: Byte oriented data types in python
Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. ctypes is basically for the interface with libraries written in C (this I read from the python docs) -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On Jan 25, 12:52 am, Martin v. Löwis mar...@v.loewis.de wrote: packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || packet_data(variable) How to construct these using python data types, as int and float have no limits and their sizes are not well defined. In Python 2.x, use the regular string type: chr(n) will create a single byte, and the + operator will do the concatenation. In Python 3.x, use the bytes type (bytes() instead of chr()). This looks really helpful thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
Ravi wrote: Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. If you read the module documentation more carefully you will see that it converts between the various native data types and character strings. Thus each native data type occupies only as many bytes as are required to store it in its native form (modulo any alignments needed). ctypes is basically for the interface with libraries written in C (this I read from the python docs) I believe it *is* the struct module you need. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Ravi ra.ravi@gmail.com wrote: Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. I don't know what you mean by returns an expanded string of the data. I do know that struct does exactly what you requested. It converts between Python objects and what is bascially a C struct where you specify the endianness of each field and what sort of packing/padding you want. I use the struct module frequenty to impliment binary, communications protocols in Python. I've used Python/struct with transport layers ranging from Ethernet (raw, TCP, and UDP) to async serial, to CAN. -- -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On Sun, Jan 25, 2009 at 7:27 AM, Ravi ra.ravi@gmail.com wrote: Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. Noo... struct really IS the choice; that is the explicit purpose of the struct library. I went and included an example too which you're not noticing :) Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin Type help, copyright, credits or license for more information. import struct data = struct.pack(BB4s, 1, 4, this) data '\x01\x04this' That's precisely six raw bytes which is exactly what you specified you needed: exactly one unsigned byte for packet type, one unsigned byte for length, and four bytes for the data after. In real life you'd probably use something besides 4s to pack what you marked as other and variable, of course, but still. Replace it with whatever your message requires. Really, this is how you do line protocols ;) Well there's other ways, but... --S -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. I don't know what you mean by returns an expanded string of the data. I do know that struct does exactly what you requested. I disagree. He has a format (type, length, value), with the value being variable-sized. How do you do that in the struct module? It converts between Python objects and what is bascially a C struct where you specify the endianness of each field and what sort of packing/padding you want. Sure. However, in the specific case, there is really no C struct that can reasonably represent the data. Hence you cannot really use the struct module. I use the struct module frequenty to impliment binary, communications protocols in Python. I've used Python/struct with transport layers ranging from Ethernet (raw, TCP, and UDP) to async serial, to CAN. Do you use it for the fixed-size parts, or also for the variable-sized data? Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote: Take a look at the struct and ctypes modules. struct is really not the choice. it returns an expanded string of the data and this means larger latency over bluetooth. I don't know what you mean by returns an expanded string of the data. I do know that struct does exactly what you requested. I disagree. He has a format (type, length, value), with the value being variable-sized. How do you do that in the struct module? You construct a format string for the value portion based on the type/length header. It converts between Python objects and what is bascially a C struct where you specify the endianness of each field and what sort of packing/padding you want. Sure. However, in the specific case, there is really no C struct that can reasonably represent the data. I don't see how that can be the case. There may not be a single C struct that can represent all frames, but for every frame you should be able to come up with a C struct that can represent that frame. Hence you cannot really use the struct module. Perhaps I don't understand his requirements, but I use the struct module for protocols with type/len/value sorts of packets. I use the struct module frequenty to impliment binary, communications protocols in Python. I've used Python/struct with transport layers ranging from Ethernet (raw, TCP, and UDP) to async serial, to CAN. Do you use it for the fixed-size parts, or also for the variable-sized data? Both. For varible size/format stuff you decode the first few bytes and use them to figure out what format/layout to use for the next chunk of data. It's pretty much the same thing you do in other languages. -- Grant -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
I disagree. He has a format (type, length, value), with the
value being variable-sized. How do you do that in the struct
module?
You construct a format string for the value portion based on
the type/length header.
Can you kindly provide example code on how to do this?
I don't see how that can be the case. There may not be a
single C struct that can represent all frames, but for every
frame you should be able to come up with a C struct that can
represent that frame.
Sure. You would normally have a struct such as
struct TLV{
char type;
char length;
char *data;
};
However, the in-memory representation of that struct is *not*
meant to be sent over the wire. In particular, the character
pointer has no meaning outside the address space, and is thus
not to be sent.
Both. For varible size/format stuff you decode the first few
bytes and use them to figure out what format/layout to use for
the next chunk of data. It's pretty much the same thing you do
in other languages.
In the example he gave, I would just avoid using the struct module
entirely, as it does not provide any additional value:
def encode(type, length, value):
return chr(type)+chr(length)+value
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On Jan 26, 2:28 am, Ravi ra.ravi@gmail.com wrote: On Jan 25, 12:52 am, Martin v. Löwis mar...@v.loewis.de wrote: packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || packet_data(variable) How to construct these using python data types, as int and float have no limits and their sizes are not well defined. In Python 2.x, use the regular string type: chr(n) will create a single byte, and the + operator will do the concatenation. In Python 3.x, use the bytes type (bytes() instead of chr()). This looks really helpful thanks! Provided that you don't take Martin's last sentence too literally :-) | Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] on win32 | p_data = babcd # Omit the b prefix if using 2.5 or earlier | p_len = len(p_data) | p_type = 3 | chr(p_type) + chr(p_len) + p_data | '\x03\x04abcd' | Python 3.0 (r30:67507, Dec 3 2008, 20:14:27) [MSC v.1500 32 bit (Intel)] on win32 | p_data = babcd | p_len = len(p_data) | p_type = 3 | bytes(p_type) + bytes(p_len) + p_data # literal translation | b'\x00\x00\x00\x00\x00\x00\x00abcd' | bytes(3) | b'\x00\x00\x00' | bytes(10) | b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' | bytes([p_type]) + bytes([p_len]) + p_data | b'\x03\x04abcd' | bytes([p_type, p_len]) + p_data | b'\x03\x04abcd' Am I missing a better way to translate chr(n) from 2.x to 3.x? The meaning assigned to bytes(n) in 3.X is interesting: 2.X: nuls = '\0' * n out_byte = chr(n) 3.X: nuls = b'\0' * n or nuls = bytes(n) out_byte = bytes([n]) Looks to me like there was already a reasonable way of getting a bytes object containing a variable number of zero bytes. Any particular reason why bytes(n) was given this specialised meaning? Can't be the speed, because the speed of bytes(n) on my box is about 50% of the speed of the * expression for n = 16 and about 65% for n = 1024. Cheers, John -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
Looks to me like there was already a reasonable way of getting a bytes object containing a variable number of zero bytes. Any particular reason why bytes(n) was given this specialised meaning? I think it was because bytes() was originally mutable, and you need a way to create a buffer of n bytes. Now that bytes() ended up immutable (and bytearray was added), it's perhaps not so useful anymore. Of course, it would be confusing if bytes(4) created a sequence of one byte, yet bytearray(4) created four bytes. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote:
You construct a format string for the value portion based on
the type/length header.
Can you kindly provide example code on how to do this?
OK, something like this to handle received data where there is
an initial 8-bit type field that is 1 for 16-bit unsigned
integers in network byte-order, 2 for 32-bit IEEE floats in
network byte-order. We'll further assume that the 'length'
field comes next as a 16 bit unsigned value in network order
and represents how many objects of the specified type follow:
dtype = ord(rawdata[0])
dcount = struct.unpack(!H,rawdata[1:3])
if dtype == 1:
fmtstr = ! + H*dcount
elif dtype == 2:
fmtstr = ! + f*dcount
rlen = struct.calcsize(fmtstr)
data = struct.unpack(fmtstr,rawdata[3:3+rlen])
leftover = rawdata[3+rlen:]
I don't see how that can be the case. There may not be a
single C struct that can represent all frames, but for every
frame you should be able to come up with a C struct that can
represent that frame.
Sure. You would normally have a struct such as
struct TLV{
char type;
char length;
char *data;
};
However, the in-memory representation of that struct is *not*
meant to be sent over the wire. In particular, the character
pointer has no meaning outside the address space, and is thus
not to be sent.
Well if it's not representing the layout of the data we're
trying to deal with, then it's irrelevent. We are talking
about how convert python objects to/from data in the
'on-the-wire' format, right?
Or isn't that what the OP is asking about?
Both. For varible size/format stuff you decode the first few
bytes and use them to figure out what format/layout to use for
the next chunk of data. It's pretty much the same thing you do
in other languages.
In the example he gave, I would just avoid using the struct module
entirely, as it does not provide any additional value:
def encode(type, length, value):
return chr(type)+chr(length)+value
Like this?
def encode(type,length,value):
... return chr(type)+chr(length)+value
...
print encode('float', 1, 3.14159)
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 2, in encode
TypeError: an integer is required
--
Grant
--
http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
dtype = ord(rawdata[0])
dcount = struct.unpack(!H,rawdata[1:3])
if dtype == 1:
fmtstr = ! + H*dcount
elif dtype == 2:
fmtstr = ! + f*dcount
rlen = struct.calcsize(fmtstr)
data = struct.unpack(fmtstr,rawdata[3:3+rlen])
leftover = rawdata[3+rlen:]
Unfortunately, that does not work in the example. We have
a message type (an integer), and a variable-length string.
So how do you compute the struct format for that?
Sure. You would normally have a struct such as
struct TLV{
char type;
char length;
char *data;
};
However, the in-memory representation of that struct is *not*
meant to be sent over the wire. In particular, the character
pointer has no meaning outside the address space, and is thus
not to be sent.
Well if it's not representing the layout of the data we're
trying to deal with, then it's irrelevent. We are talking
about how convert python objects to/from data in the
'on-the-wire' format, right?
Right: ON-THE-WIRE, not IN MEMORY. In memory, there is a
pointer. On the wire, there are no pointers.
Like this?
def encode(type,length,value):
... return chr(type)+chr(length)+value
...
print encode('float', 1, 3.14159)
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 2, in encode
TypeError: an integer is required
No:
py CONNECT_REQUEST=17
py payload=call me
py encode(CONNECT_REQUEST, len(payload), payload)
'\x11\x07call me'
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote: dtype = ord(rawdata[0]) dcount = struct.unpack(!H,rawdata[1:3]) if dtype == 1: fmtstr = ! + H*dcount elif dtype == 2: fmtstr = ! + f*dcount rlen = struct.calcsize(fmtstr) data = struct.unpack(fmtstr,rawdata[3:3+rlen]) leftover = rawdata[3+rlen:] Unfortunately, that does not work in the example. We have a message type (an integer), and a variable-length string. So how do you compute the struct format for that? I'm confused. Are you asking for an introductory tutorial on programming in Python? Right: ON-THE-WIRE, not IN MEMORY. In memory, there is a pointer. On the wire, there are no pointers. I don't understand your point. py CONNECT_REQUEST=17 py payload=call me py encode(CONNECT_REQUEST, len(payload), payload) '\x11\x07call me' If all your data is comprised of 8-bit bytes, then you don't need the struct module. -- Grant -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
Unfortunately, that does not work in the example. We have a message type (an integer), and a variable-length string. So how do you compute the struct format for that? I'm confused. Are you asking for an introductory tutorial on programming in Python? Perhaps. I honestly do not know how to deal with variable-sized strings in the struct module in a reasonable way, and thus believe that this module is incapable of actually supporting them (unless you use inappropriate trickery). However, as you keep claiming that the struct module is what should be used, I must be missing something about the struct module. I don't understand your point. py CONNECT_REQUEST=17 py payload=call me py encode(CONNECT_REQUEST, len(payload), payload) '\x11\x07call me' If all your data is comprised of 8-bit bytes, then you don't need the struct module. Go back to the original message of the OP. It says # I have following packet format which I have to send over Bluetooth. # packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || # packet_data(variable) So yes, all his date is comprised of 8-bit bytes, and yes, he doesn't need the struct module. Hence I'm puzzled why people suggest that he uses the struct module. I think the key answer is use the string type, it is appropriate to represent byte oriented data in python (also see the subject of this thread) Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote: Unfortunately, that does not work in the example. We have a message type (an integer), and a variable-length string. So how do you compute the struct format for that? I'm confused. Are you asking for an introductory tutorial on programming in Python? Perhaps. I honestly do not know how to deal with variable-sized strings in the struct module in a reasonable way, and thus believe that this module is incapable of actually supporting them (unless you use inappropriate trickery). It deals with variable sized fields just fine: dtype = 18 dlength = 32 format = !BB%ds % dlength rawdata = struct.pack(format, (dtype,dlength,data)) However, as you keep claiming that the struct module is what should be used, I must be missing something about the struct module. http://docs.python.org/library/struct.html I don't understand your point. py CONNECT_REQUEST=17 py payload=call me py encode(CONNECT_REQUEST, len(payload), payload) '\x11\x07call me' If all your data is comprised of 8-bit bytes, then you don't need the struct module. Go back to the original message of the OP. It says # I have following packet format which I have to send over Bluetooth. # packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || # packet_data(variable) So yes, all his date is comprised of 8-bit bytes, He doesn't specify what format the packet_data is, and we all assumed he needed to handle conversion of various data types to/from raw byte-strings. and yes, he doesn't need the struct module. Hence I'm puzzled why people suggest that he uses the struct module. We all assumed that packet_data might contain values of various types such as 16 or 32 bit integers, floating point values -- that packet_data was not solely arbitrary-length strings of 8-bit bytes. I think the key answer is use the string type, it is appropriate to represent byte oriented data in python (also see the subject of this thread) I, for one, interpreted byte-oriented to mean that the data was received/sent as blocks of bytes but needed to be converted into other data types. If the data really is just strings of bytes, and it's sent as strings of bytes, then I have no idea what the OP was asking, since there's nothing that needs to be done with the data. -- Grant -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
It deals with variable sized fields just fine: dtype = 18 dlength = 32 format = !BB%ds % dlength rawdata = struct.pack(format, (dtype,dlength,data)) I wouldn't call this just fine, though - it involves a % operator to even compute the format string. IMO, it is *much* better not to use the struct module for this kind of problem, and instead rely on regular string concatenation. Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On 2009-01-25, Martin v. Löwis mar...@v.loewis.de wrote: It deals with variable sized fields just fine: dtype = 18 dlength = 32 format = !BB%ds % dlength rawdata = struct.pack(format, (dtype,dlength,data)) I wouldn't call this just fine, though - it involves a % operator to even compute the format string. IMO, it is *much* better not to use the struct module for this kind of problem, and instead rely on regular string concatenation. If all you need to do is concatenate strings, then you're correct, there's no advantage to using struct or ctypes. If you need a generic way to deal with arbitrary data types, then that's what the struct and ctypes modules are designed to do. The protocols I've implemented always required the ability to deal with integers greater than 8 bits wide as well as various other data types. -- Grant -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
However, as you keep claiming that the struct module is what should be used, I must be missing something about the struct module. You seem to be focusing overly on the C Struct part of the description of what the struct module does, instead of the part where it says, packed binary data and This can be used in handling binary data stored in files or from network connections The struct module is very useful when generating or interpreting a stream of binary data which has a set structured format, in bytes and with set kinds of data types. # I have following packet format which I have to send over Bluetooth. # packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || # packet_data(variable) So yes, all his date is comprised of 8-bit bytes, and yes, he doesn't need the struct module. Hence I'm puzzled why people suggest that he uses the struct module. I think the key answer is use the string type, it is appropriate to represent byte oriented data in python (also see the subject of this thread) Everyone *is* saying to use the string type: the struct module doesn't produce structs.. it produces strings. The question is how to convert from Python types to byte-representations of said types in a string: you are saying (for example) that for shorts someone should use chr(1) etc. What if he needs to then send a two byte int from a Python int? Just because he said his message was about a fairly simple requirement at the moment, it doesn't mean that's the end of his problem. Everyone else seems to have assumed that it'll advance as he supports more of this protocol: that typelengthvariable data is apt to take on more specific struct format messages based on type. So you might have a connection request format that's pre-defined -- your example above -- or another which needs to pass say, a 2 byte int instead. The struct module easily accomplishes this: all together in one place. Instead of having to figure out how to calculate the 2 byte storage of an int down the road, or a 4 byte integer, etc. Not to mention endianness. If the OP never ever has to do anything but the one specific problem he's having right now (and this isn't just an example he's making) with really really simple data: two byte-ints and then some characters, then sure using chr() and string concatenation is probably fine. But struct works -perfectly- fine for variable data and as he extends his support for the protocol, is quite likely to be very useful to him: that's why we're suggesting it. IMHO. I'd rather spend some trival time using string concatination (or the % operator) generating format strings for struct then have to write code for int_to_2byte_int and int_to_4byte_int and change_to_bigendian for when chr() doesn't satisfy the conversation needs enough. Yeah, strings are exactly how he can and should store the byte oriented data in Python 2.x. Using concatination and slicing + conversion to read and parse a byte stream coming off a line protocol... if you want to. But there's tools that are there that make it all alot easier too. The struct module for example. --Stephen -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
On Jan 26, 10:53 am, Martin v. Löwis mar...@v.loewis.de wrote:
It deals with variable sized fields just fine:
dtype = 18
dlength = 32
format = !BB%ds % dlength
rawdata = struct.pack(format, (dtype,dlength,data))
I wouldn't call this just fine, though - it involves
a % operator to even compute the format string. IMO,
it is *much* better not to use the struct module for this
kind of problem, and instead rely on regular string
concatenation.
IMO, it would be a good idea if struct.[un]pack supported a variable *
length operator that could appear anywhere that an integer constant
could appear, as in C's printf etc and Python's % formatting:
dlen = len(data)
rawdata = struct.pack(!BB*s, dtype, dlen, dlen, data)
# and on the other end of the wire:
dtype, dlen = struct.unpack(!BB, rawdata[:2])
data = struct.unpack(!*s, rawdata[2:], dlen)
# more than 1 count arg could be used if necessary
# *s would return a string
# *B, *H, *I, etc would return a tuple of ints in (3.X-speak)
I've worked with variable-length data that looked like
len1, len2, len3, data1, data2, data3
and the * gadget would have been very handy:
len1, len2, len3 = unpack('!BBB', raw[:3])
data1, data2, data3 = unpack('!*H*i*d', raw[3:], len1, len2, len3)
Note the semantics of '!*H*i*d' would be different from '!8H2i7d'
because otherwise you'd need to do:
bundle = unpack('!*H*i*d', raw[3:], len1, len2, len3)
data1 = bundle[:len1]
data2 = bundle[len1:len1+len2]
data3 = bundle[len1+len2:]
--
http://mail.python.org/mailman/listinfo/python-list
Byte oriented data types in python
I have following packet format which I have to send over Bluetooth. packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || packet_data(variable) How to construct these using python data types, as int and float have no limits and their sizes are not well defined. -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
I have following packet format which I have to send over Bluetooth. packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || packet_data(variable) How to construct these using python data types, as int and float have no limits and their sizes are not well defined. Check out the struct module. You want something like: data = struct.pack(BB4s, 1, 4, this) It returns a string of bytes according to the specification -- B is unsigned byte, and 4s is a string of 4 characters. The 1 feeds into the first byte, the 4 into the second, etc. --Stephen -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || packet_data(variable) How to construct these using python data types, as int and float have no limits and their sizes are not well defined. In Python 2.x, use the regular string type: chr(n) will create a single byte, and the + operator will do the concatenation. In Python 3.x, use the bytes type (bytes() instead of chr()). Regards, Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: Byte oriented data types in python
Ravi packet_type (1 byte unsigned) || packet_length (1 byte unsigned) || Ravi packet_data(variable) Ravi How to construct these using python data types, as int and float have Ravi no limits and their sizes are not well defined. Take a look at the struct and ctypes modules. -- Skip Montanaro - s...@pobox.com - http://smontanaro.dyndns.org/ -- http://mail.python.org/mailman/listinfo/python-list
