Hi Julian,

I am currently not aware of a protocol that mixes the endianness. I have heard 
that there are some modbus implementations that send the payload with different 
endianness for some implementations, but we shouldn't handle that on mspec 
level. Because then we would have super-complex mspec and/or different variants 
of the driver. Here I would prefer to have connection string arguments telling 
the drivers how to process the payload if this differs from the default.

And yes ... bringing the BE/LE into the code generation would make the code 
generation code more complex, the generated code slightly and bloat the mspec 
format (in my opinion)

And seeing that Etienne was able to implement the EIP mspec without much help 
(which was the first real BE protocol we have) seems to prove that it is not 
too complex to understand if you rid yourself of this idea that a "byte" is 
something different than an signed or unsigned 8 bit integer value.

I usually make a "byte" a "int 8" even if I think an "uint 8" would be more 
correct because of the convenience, that in Java a byte is an 8 bit signed 
integer which makes processing it simpler.


Am 14.04.20, 12:26 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>:

    as far as I see it there are technically two ways to achieve the change 
between BE / LE.
    One ist he one suggested by Lukasz, i.e. always specifying byte order for 
each read / write item in mspec.
    The other one ist o leave it out of mspec and only define it in the 
ByteReader / Writer (similar to how java or netty do it with the ByteBuffer).
    I cannot judge all the implications oft he first approach, the second is 
definetly more pragmatic (and keeps things orthogonal).
    The only situation where we would need this is a protocol which mixes byte 
orderings. Do we have such a protocol?
    Otherwise I am with Chris and would try to stay pragmatic.
    For me it looks like this would mean quite a bit of work to implement this 
(not speaking about additional complexity that could be introduced).
    Is my summary correct so far?
    Am 14.04.20, 10:52 schrieb "Łukasz Dywicki" <l...@code-house.org>:
        Hey Christian,
        The problem I face is quite simple. State type in mspec i declared as a
        bunch of bits. Type length is fixed, however not available anywhere in
        mspec or code generator to re-arrange bytes upfront. All we have exposed
        at reader/write level is read/write bit.
        To be fair LE/BE support in read/write buffers is limited just to
        numbers. There is no such support for raw bytes or bits, cause for that
        you need to declare a length of LE/BE sequence.
        I would love if I could just declare State as '2 byte little endian' so
        it would be read properly upfront and parsed with no changes in
        generated code, however I'm not sure how to do it and where. That's why
        I'm playing with different things described in earlier mail.
        Since all type handling is general I am just afraid of more complicated
        scenarios where we have variable length structures such as arrays.
        Best regards,
        On 14.04.2020 09:41, Christofer Dutz wrote:
        > Hi Lukasz,
        > I am not sure I am understanding the problems you are facing. We 
already have LE and BE protocols.
        > For example EIP is LE and the rest is generally BE. It seems that 
ADS/AMS is also LE. 
        > mspec doesn't even know about endianness.
        > Up till now the endianness doesn't have an effect on bit-fields or 
single-bit ints. 
        > It only starts to affect if a field goes from one byte to the next, 
which is usually for (u)int and floating point values.
        > That's why we have created the Read/WriteBuffers to set their 
endianness in the constructor.
        > So if you're creating a driver for ADS/AMS which is LE, then you 
write the mspec according to the sequence the information is located in the 
transferred bytes and have the Read/WriteBuffer handle the endianness issue.
        > I do see a problem when there are drivers that use mixed endianness, 
but we have still to encounter such a protocol.
        > So I have to admit that I don't like any of the mspec changes you 
proposed, as I think you are just not using the tools we have the right way.
        > Chris
        > Am 14.04.20, 00:32 schrieb "Łukasz Dywicki" <l...@code-house.org>:
        >     Hey Niclas,
        >     I realized how old the old things are when I started preparing
        >     automation training for mere mortals and got into history of 
frames and
        >     even cabling. Mr. Modbus and EIA-485 is definitely older than I. 
        >     Getting back to the point - yes. I been thinking how to address 
the byte
        >     order in effective way. Here are two approaches I have for now:
        >     A) My initial attempt is just a temporary buffer which is then 
        >     in reverse order to caller. For reading it is similar - just 
getting N
        >     bytes in reversed order. The hard part is.. knowing N. I had to 
add a
        >     static calculation in order to allocate valid buffer sizes. I 
tend to
        >     work but I'm not happy with this approach cause it involves 
additional work.
        >     B) Second idea I've got is really simple and relies on code 
        >     We know in which order fields are coming. Here I'm referring to a 
        >     field which is just bunch of bits. If we would group fields in 
bytes and
        >     generate code in reverse order then it has chance to work. 
        >     for that - ability to know basic field sizes upfront.
        >     C) Try to combine above with bit-io or Read/WriteBuffers as these 
        >     places which know actual position and state of buffers which are 
        >     read/written.
        >     Now, getting to two cases which are a problem. CommandId and 
State. So
        >     with command id situation is simple as it is declared as enum and 
it is
        >     read as uint. We know size upfront and can generate valid method 
        >     (readIntLE).
        >     [enum uint 16 little endian 'CommandId'
        >         ['0x00' INVALID]
        >         ['0x01' ADS_READ_DEVICE_INFO]
        >         ['0x02' ADS_READ]
        >         ['0x03' ADS_WRITE]
        >         ['0x04' ADS_READ_STATE]
        >         ['0x05' ADS_WRITE_CONTROL]
        >         ['0x06' ADS_ADD_DEVICE_NOTIFICATION]
        >         ['0x07' ADS_DELETE_DEVICE_NOTIFICATION]
        >         ['0x08' ADS_DEVICE_NOTIFICATION]
        >         ['0x09' ADS_READ_WRITE]
        >     ]
        >     Second candidate is what I'm stuck right now sniping next cycles 
        >     problems. So in case of State we have complex type composed from 2
        >     bytes. A note here - instead of two bytes we might have a variable
        >     length type which includes array or other variable section.
        >     [type little endian 'State'
        >         [simple     bit 'broadcast'             ]
        >         [reserved   int 7 '0x0'                 ]
        >         [simple     bit 'initCommand'           ]
        >         [simple     bit 'updCommand'            ]
        >         [simple     bit 'timestampAdded'        ]
        >         [simple     bit 'highPriorityCommand'   ]
        >         [simple     bit 'systemCommand'         ]
        >         [simple     bit 'adsCommand'            ]
        >         [simple     bit 'noReturn'              ]
        >         [simple     bit 'response'              ]
        >     ]
        >     The order of reading big endian encoded data to impose little 
        >     shift would be (please correct me if I'm wrong):
        >     1) init
        >     2) udp
        >     3) add timestamp
        >     4) priority
        >     5) system
        >     6) ads
        >     7) noreturn
        >     8) response (end of byte 1)
        >     9) broadcast
        >     10) reserved (end of byte )
        >     We can do same trick for writing, by re-arranging fields. By this 
way we
        >     avoid any additional byte level operations.
        >     Overall trouble with generated driver is to declare "how much" 
        >     should be read and interpreted. We have precise size information 
at the
        >     runtime - due to length fields, we can leverage it at generation 
        >     but then we won't be able to cover all cases.
        >     I would love to keep it simple and do not break things thus I 
need your
        >     advice on how to approach this problem in a valid way.
        >     Cheers,
        >     Łukasz
        >     On 13.04.2020 03:26, Niclas Hedhman wrote:
        >     > <anecdotal-rant>
        >     > For us who were around and shaping the protocols in the 1980s, 
and people
        >     > before us (and before standards like RS-232), a lot of the 
        >     > came out of "observation of implementation we managed to get to 
        >     > rather than "implement this spec". A lot was due to extreme 
        >     > constraints (in my case, multi-tasking operating system, serial 
        >     > 187kbps, interpreted programming language with floating point 
ops and user
        >     > applications in 2kB RAM and 8kB EPROM) and a general lack of 
        >     > like what other people were doing, sharing experiences and so 
        >     > 
        >     > And there were many "innovative" ways to squeeze just a little 
bit extra
        >     > out of the hardware, resulting in "hard to understand" 
consequences. Bit
        >     > packing was a typical one, multiple functions packed into a 
single byte.
        >     > Look at page 14 in 
        >     > and read up on "UART Enahanced Mode", and we used this, i.e. 9 
bits, no
        >     > parity and clever use of address and mask to create a 
slave-to-slave direct
        >     > protocol, where the master's role was to signal which slave 
"owned" the
        >     > cable. Yeah, in that 8kB ROM limitation (I think protocol was 
about 1kB
        >     > ROM) and something like 150 bytes RAM for comm protocol.
        >     > 
        >     > Could you implement a compatible device to this with PLC4X and 
        >     > hardware (i.e. no 8031/32 co-processor)? Possibly but 
bit-banging is needed
        >     > to support the 9bit data (+start and stop bits) and an awful 
lot of CPU
        >     > cycles on something that was automatic on one of the slowest 
        >     > microcontroller ever.
        >     > </anecdotal-rant>
        >     > 
        >     > My point was only to highlight that some of the strange things 
you see in
        >     > protocols today, have its roots in pre-standardization days. 
Today no one
        >     > would go down that route, because the hardware cost nothing now 
(8031  +
        >     > 8kB EPROM + 2kB static RAM + battery backup => ~$50 in 1983's 
currency) and
        >     > longevity of software is more important.
        >     > 
        >     > Cheers
        >     > Niclas
        >     > 
        >     > 
        >     > On Sun, Apr 12, 2020 at 10:10 PM Christofer Dutz 
        >     > wrote:
        >     > 
        >     >> Hi Lukasz,
        >     >>
        >     >> I think it really gets tricky when using BE and having some 
        >     >> ... I remember in the Firmata protocol there were some 
bitmasks and then 10
        >     >> bit uint as BE ... not it really got tricky as the specs were 
written from
        >     >> a point of view: You read 16 bits BE and then the first6 bits 
mean XYZ
        >     >> instead of describing how the bits actually travel over the 
        >     >>
        >     >> Chris
        >     >>
        >     >>
        >     >>
        >     >> Am 11.04.20, 01:21 schrieb "Łukasz Dywicki" 
        >     >>
        >     >>     I've made some progress with topic by modyfing mspec and 
        >     >>     'little endian' flag on fields. This moved me further to 
next issue -
        >     >>     which is whole type encoded little endian.
        >     >>
        >     >>     In ADS driver such type is State, which has 2 bytes and 
uses 8 bits for
        >     >>     various flags.
        >     >>     There are two cases which require different approach - 
reading and
        >     >>     writing. So for reading we need to swap N bytes based on 
type length.
        >     >>     For writing we need to alocate buffer for N bytes and swap 
them before
        >     >>     writing.
        >     >>
        >     >>     I am stuck now with freemaker templates and bit-io.
        >     >>
        >     >>     Cheers,
        >     >>     Łukasz
        >     >>
        >     >>
        >     >>
        >     >>     On 10.04.2020 17:57, Łukasz Dywicki wrote:
        >     >>     > I am doing some tests of ADS serialization.
        >     >>     >
        >     >>     > I've run into some troubles with payload which is 
generated with new
        >     >>     > driver. I'm not sure if that's my fault or generated 
        >     >>     >
        >     >>     > I did a verification of what Wireshark shows and how ads 
        >     >> are
        >     >>     > parsed. There is a gap I think. For example ams port 
number 1000
        >     >>     > (0x1027) is read as 4135.
        >     >>     >
        >     >>     > Obviously I used wrong structures while implementing 
protocol logic
        >     >> in
        >     >>     > first place, but now I am uncertain of how fields are 
encoded. How we
        >     >>     > mark field as little endian when rest of payload is big 
endian? Do we
        >     >>     > have `uint_le`?
        >     >>     >
        >     >>     > As far I remember route creation logic I was tracking 
last week used
        >     >>     > combination of LE and BE.
        >     >>     >
        >     >>     > Best regards,
        >     >>     > Łukasz
        >     >>     >
        >     >>
        >     >>
        >     >>
        >     > 

Reply via email to