Re: [OpenWrt-Devel] [RFC] libubox/binary.h design proposal

[Javier Domingo Cansino]

Hello,

I have been asked for examples and use cases. Github repo has been updated.

I do also have doubts on if it's worth it. If a library like this existed I
would be definitely using it, make conversions, does take care of
alignment, can manage unaligned data and endianness. etc.

But I am not really sure if someone apart from me would find it interesting
(long time ago I haven't seen a protocol unaligned with different
endianess).

Waiting for your feedback!

/*
 * Syntax for the function is as follows.
 *   %[p][n][a][lb][01r]{i,y,w,d,q,s}
 *
 *   All the characters that don't follow this will be treated as raw chars
 *   to be written as they are.
 *
 * Data type
 *   * i - bit
 *   * y - byte
 *   * w - 2 byte word
 *   * d - 4 byte word
 *   * q - 8 byte word
 *   * s - string without termination (use strlen()+1 in quantity to null)
 *
 * Data value
 *   * 0 - fill the specified space with zeros
 *   * 1 - fill the specified space with ones
 *   * r - fill the specified space with random data
 *
 *   String data type is not valid in this case. When quantity is also
 *   specified, use l/b to separate: %5b0 -> 5 bits of value 0
 *
 * Endianess. No conversion by default
 *   * l - little endian
 *   * b - big endian
 *
 * Alignment. No alignment by default
 *   * a - align this to it's datatype
 * bits are aligned to byte
 *
 * Quantity. One by default
 *   * n - number of same datatype (placed together)
 * this denotes length of string, padded with 0
 *
 * Pointer. Default everything is passed by value
 *   * p - specify that passed parameter is a pointer to value(s)
 *
 *   Obviously, when quantity is specified, and datatype is not bits, p
 *   is required.
 *
 * Some examples:
 *   * %4lw - 4 little endian 2 byte word
 *   * %2i  - 2 bits 'ab' from value b'00ab'
 *   * %2bi - 2 bits 'ab' from value b'00ba'
 *   * %2li - 2 bits 'ab' from value b'ab00'
 *   * %5y
 *
 * Calling functions example:
 * int32_t dw[]=[1,2,3,4]
 *
 * v = writebv("%lw%2i%ay%p4d", 2047, 3, 'n', dw);
 *
 * in a big endian computer
 * // *v = 0xff07c06e0001000200030004
 *
 * struct my_data_t {
 *   uint16_t a;
 *   uint8_t b:2;
 *   uint8_t unused1:6;
 *   uint8_t c;
 *   uint32_t d[4];
 * } *my_data;
 * char letter;
 *
 * // If we want to have all extracted from the blob
 * my_data = readbv(v, "%lw%2i%ay%p4d");
 *
 * // If we just want to extract the char
 * readb(v, "%0w%2l0i%ay%4l0d", );
 *
 * // We have aligned data, but if we removed the 'a' from %ay, we
 * wouldn't have unaligned data and this would still work.
 *
 */
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] [RFC] libubox/binary.h design proposal

[Javier Domingo Cansino]

After feedback in IRC, I will drop this idea, and try to go the kernel way,
supporting endianess annotated structures with sparse rules checking.

https://lwn.net/Articles/205624/

The motivation is that it's really difficult to keep in sync all the format
strings, and although there may be ways to deal with it, I rather prefer
this other design.

On Mon, Oct 5, 2015 at 10:25 AM Javier Domingo Cansino 
wrote:

> Hello,
>
> I have been asked for examples and use cases. Github repo has been updated.
>
> I do also have doubts on if it's worth it. If a library like this existed
> I would be definitely using it, make conversions, does take care of
> alignment, can manage unaligned data and endianness. etc.
>
> But I am not really sure if someone apart from me would find it
> interesting (long time ago I haven't seen a protocol unaligned with
> different endianess).
>
> Waiting for your feedback!
>
> /*
>  * Syntax for the function is as follows.
>  *   %[p][n][a][lb][01r]{i,y,w,d,q,s}
>  *
>  *   All the characters that don't follow this will be treated as raw chars
>  *   to be written as they are.
>  *
>  * Data type
>  *   * i - bit
>  *   * y - byte
>  *   * w - 2 byte word
>  *   * d - 4 byte word
>  *   * q - 8 byte word
>  *   * s - string without termination (use strlen()+1 in quantity to null)
>  *
>  * Data value
>  *   * 0 - fill the specified space with zeros
>  *   * 1 - fill the specified space with ones
>  *   * r - fill the specified space with random data
>  *
>  *   String data type is not valid in this case. When quantity is also
>  *   specified, use l/b to separate: %5b0 -> 5 bits of value 0
>  *
>  * Endianess. No conversion by default
>  *   * l - little endian
>  *   * b - big endian
>  *
>  * Alignment. No alignment by default
>  *   * a - align this to it's datatype
>  * bits are aligned to byte
>  *
>  * Quantity. One by default
>  *   * n - number of same datatype (placed together)
>  * this denotes length of string, padded with 0
>  *
>  * Pointer. Default everything is passed by value
>  *   * p - specify that passed parameter is a pointer to value(s)
>  *
>  *   Obviously, when quantity is specified, and datatype is not bits, p
>  *   is required.
>  *
>  * Some examples:
>  *   * %4lw - 4 little endian 2 byte word
>  *   * %2i  - 2 bits 'ab' from value b'00ab'
>  *   * %2bi - 2 bits 'ab' from value b'00ba'
>  *   * %2li - 2 bits 'ab' from value b'ab00'
>  *   * %5y
>  *
>  * Calling functions example:
>  * int32_t dw[]=[1,2,3,4]
>  *
>  * v = writebv("%lw%2i%ay%p4d", 2047, 3, 'n', dw);
>  *
>  * in a big endian computer
>  * // *v = 0xff07c06e0001000200030004
>  *
>  * struct my_data_t {
>  *   uint16_t a;
>  *   uint8_t b:2;
>  *   uint8_t unused1:6;
>  *   uint8_t c;
>  *   uint32_t d[4];
>  * } *my_data;
>  * char letter;
>  *
>  * // If we want to have all extracted from the blob
>  * my_data = readbv(v, "%lw%2i%ay%p4d");
>  *
>  * // If we just want to extract the char
>  * readb(v, "%0w%2l0i%ay%4l0d", );
>  *
>  * // We have aligned data, but if we removed the 'a' from %ay, we
>  * wouldn't have unaligned data and this would still work.
>  *
>  */
>
>
>
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

[OpenWrt-Devel] [RFC] libubox/binary.h design proposal

[Javier Domingo Cansino]

Hello,

I asked Friday on IRC how to write blobs, I was suggested using
blob_raw_put from libubox/blob.h, but as I have to implement a binary
protocol that uses different endianess, non aligned data etc., I think the
best solution is creating a set of helper functions to write/read blobs.

As this is quite generic library, there might already exist some work done
that I am missing (please link!). I would love to have feedback from this
design before typing code.

I have started my work on github[1], which is basically a header file with
the following comment and some prototype ideas. The syntax I have thought
is described there, but it basically follows a printf syntax, allowing raw
chars to be written.

Cheers!

/*
 * Syntax for the function is as follows.
 *   %[n][a][lb][01r]{i,y,w,d,q,s}
 *
 *   All the characters that don't follow this will be treated as raw chars
 *   to be written as they are.
 *
 * Data type
 *   * i - bit
 *   * y - byte
 *   * w - 2 byte word
 *   * d - 4 byte word
 *   * q - 8 byte word
 *   * s - string without termination (use strlen()+1 in quantity to null)
 *
 * Data value
 *   * 0 - fill the specified space with zeros
 *   * 1 - fill the specified space with ones
 *   * r - fill the specified space with random data
 *
 *   String data type is not valid in this case
 *
 * Endianess. No conversion by default
 *   * l - little endian
 *   * b - big endian
 *
 * Alignment. No alignment by default
 *   * a - align this to it's datatype
 * bits are aligned to byte
 *
 * Quantity. One by default
 *   * n - number of same datatype (placed together)
 * this denotes length of string, padded with 0
 *
 * Some examples:
 *   * %4lw - 4 little endian 2 byte word
 *   * %2i  - 2 bits 'ab' from value b'00ab'
 *   * %2bi - 2 bits 'ab' from value b'00ba'
 *   * %2li - 2 bits 'ab' from value b'ab00'
 *
 */


[1] Github branch: https://github.com/txomon/libubox
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] [RFC] libubox/binary.h design proposal

[Etienne Champetier]

Hi,

2015-10-04 22:47 GMT+02:00 Javier Domingo Cansino :

> Hello,
>
> I asked Friday on IRC how to write blobs, I was suggested using
> blob_raw_put from libubox/blob.h, but as I have to implement a binary
> protocol that uses different endianess, non aligned data etc., I think the
> best solution is creating a set of helper functions to write/read blobs.
>

I don't know what you really want to do, but have you looked at protobuf?
https://en.wikipedia.org/wiki/Protocol_Buffers



>
> As this is quite generic library, there might already exist some work done
> that I am missing (please link!). I would love to have feedback from this
> design before typing code.
>
> I have started my work on github[1], which is basically a header file with
> the following comment and some prototype ideas. The syntax I have thought
> is described there, but it basically follows a printf syntax, allowing raw
> chars to be written.
>
> Cheers!
>
> /*
>  * Syntax for the function is as follows.
>  *   %[n][a][lb][01r]{i,y,w,d,q,s}
>  *
>  *   All the characters that don't follow this will be treated as raw chars
>  *   to be written as they are.
>  *
>  * Data type
>  *   * i - bit
>  *   * y - byte
>  *   * w - 2 byte word
>  *   * d - 4 byte word
>  *   * q - 8 byte word
>  *   * s - string without termination (use strlen()+1 in quantity to null)
>  *
>  * Data value
>  *   * 0 - fill the specified space with zeros
>  *   * 1 - fill the specified space with ones
>  *   * r - fill the specified space with random data
>  *
>  *   String data type is not valid in this case
>  *
>  * Endianess. No conversion by default
>  *   * l - little endian
>  *   * b - big endian
>  *
>  * Alignment. No alignment by default
>  *   * a - align this to it's datatype
>  * bits are aligned to byte
>  *
>  * Quantity. One by default
>  *   * n - number of same datatype (placed together)
>  * this denotes length of string, padded with 0
>  *
>  * Some examples:
>  *   * %4lw - 4 little endian 2 byte word
>  *   * %2i  - 2 bits 'ab' from value b'00ab'
>  *   * %2bi - 2 bits 'ab' from value b'00ba'
>  *   * %2li - 2 bits 'ab' from value b'ab00'
>  *
>  */
>
>
> [1] Github branch: https://github.com/txomon/libubox
>
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>
>
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] [RFC] libubox/binary.h design proposal

[Javier Domingo Cansino]

Hi Etienne,

Sorry if I didn't express myself correctly. I meant that I have to create
code to communicate through that protocol. I am not designing a protocol.
Anyway, I was more interested on feedback from similar libraries or about
the design of the library.

Cheers!

On Sun, Oct 4, 2015 at 10:38 PM Etienne Champetier <
champetier.etie...@gmail.com> wrote:

> Hi,
>
> 2015-10-04 22:47 GMT+02:00 Javier Domingo Cansino :
>
>> Hello,
>>
>> I asked Friday on IRC how to write blobs, I was suggested using
>> blob_raw_put from libubox/blob.h, but as I have to implement a binary
>> protocol that uses different endianess, non aligned data etc., I think the
>> best solution is creating a set of helper functions to write/read blobs.
>>
>
> I don't know what you really want to do, but have you looked at protobuf?
> https://en.wikipedia.org/wiki/Protocol_Buffers
>
>
>
>>
>> As this is quite generic library, there might already exist some work
>> done that I am missing (please link!). I would love to have feedback from
>> this design before typing code.
>>
>> I have started my work on github[1], which is basically a header file
>> with the following comment and some prototype ideas. The syntax I have
>> thought is described there, but it basically follows a printf syntax,
>> allowing raw chars to be written.
>>
>> Cheers!
>>
>> /*
>>  * Syntax for the function is as follows.
>>  *   %[n][a][lb][01r]{i,y,w,d,q,s}
>>  *
>>  *   All the characters that don't follow this will be treated as raw
>> chars
>>  *   to be written as they are.
>>  *
>>  * Data type
>>  *   * i - bit
>>  *   * y - byte
>>  *   * w - 2 byte word
>>  *   * d - 4 byte word
>>  *   * q - 8 byte word
>>  *   * s - string without termination (use strlen()+1 in quantity to null)
>>  *
>>  * Data value
>>  *   * 0 - fill the specified space with zeros
>>  *   * 1 - fill the specified space with ones
>>  *   * r - fill the specified space with random data
>>  *
>>  *   String data type is not valid in this case
>>  *
>>  * Endianess. No conversion by default
>>  *   * l - little endian
>>  *   * b - big endian
>>  *
>>  * Alignment. No alignment by default
>>  *   * a - align this to it's datatype
>>  * bits are aligned to byte
>>  *
>>  * Quantity. One by default
>>  *   * n - number of same datatype (placed together)
>>  * this denotes length of string, padded with 0
>>  *
>>  * Some examples:
>>  *   * %4lw - 4 little endian 2 byte word
>>  *   * %2i  - 2 bits 'ab' from value b'00ab'
>>  *   * %2bi - 2 bits 'ab' from value b'00ba'
>>  *   * %2li - 2 bits 'ab' from value b'ab00'
>>  *
>>  */
>>
>>
>> [1] Github branch: https://github.com/txomon/libubox
>>
>> ___
>> openwrt-devel mailing list
>> openwrt-devel@lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>
>>
>
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel