Hi Lukasz, Well, mspec wasn't built for parser arguments ... it was built for providing the model/parsers/serializers.
I also probably wouldn't want to mix in multiple things into the mspec and make it an unmanageable beast. I really liked the way I separated thing in that POC I did with Daffodil and SCXML ... cause there you had syntax in Daffodil and Semantics in SCXML. In general, it even looks if something similar to my SCXML description might even be a good way to go for us. But I think we should not mix up and merge steps when it comes to implementing them ... each step will require an effort to reach ... so I would really like to do things in the steps Sebastian and I proposed, as they reduce the effort to reach the next level and when having reached that as a checkpoint, we can continue to the next. In the end I would love to have all the definition of steps 2-5 all be reflected in ideally one document so in the end we would end up with a syntax.mspec and a semantic.xml Getting encryption into mspec really might be a challenge ... cause in reality it would sort of create wrappers around the Read-/WriteBuffers ... so the plumbing would probably be problematic. Right now, I would suggest that you simply split your mspec into two logical parts ... the one reading the encrypted inner data and then the part that parses the decrypted data ... so you would ProtocolXUnencrypedParse -> decryption logic -> ProtocolXEncrypedParse Encryption isn't just something you sprinkle on something ... I haven't come actross two protocols that do it the same way ... this would always be very tricky to get right (in mspec) Chris -----Ursprüngliche Nachricht----- Von: Łukasz Dywicki <[email protected]> Gesendet: Mittwoch, 21. April 2021 11:05 An: [email protected] Betreff: Re: AW: Next evolution of code generation It was a real pleasure to read your mail Sebastian. Looking at points you raised: Stage 2) Message instances This is very much needed cause of protocols with wrappers who look like a matryoshka. Avoiding the need to process wrappers on read/write ends will greatly simplify driver code. Obviously writing from our end is easier, it is a bit harder on receiving side where same wrapper/wrappers are used for multiple payloads. Stage 3,4) We can generate an interaction for a ping-pong/conversations I merge these two points cause they are very close to each other and have request/response semantics. This semantics is sometimes hard to express without additional information. Here I mean mainly exchange id, segmentation of long packets (does the exchange id change within segmentation) and and assembling data back. There are multiple strategies in this regard. The easiest is just merging a byte[] split across few frames while others involve extra logic (ie. magic markers at the end of M-Bus frame). Stage 5) Properties and options I could throw here few additional "common" types such as Ethernet frame, IPv4 addresses and MACs which get distributed across many protocols. Obvious limitation is insufficient expressiveness of mspec for such kind of constructions. I guess that most of that work is intended for C to simplify driver URI handling, however I would opt for creation of canonical representations of parsed types first. Stage 6) Domination I believe if we get above points sorted within the code then we will still need make docs for it. ;-) There is one more step which I would like to mention which is encryption. Given that you brought a lot of stuff related to mspec improvements I need to throw also encryption as an need. There are cases (such as WM-Bus) where payload is wrapped with AES and needs extra step to lookup encryption key before parsing payload. I see a point in all steps you raised as they will allow to better balance maintenance between languages I do not look at. Especially the conversation part is a burden which is pretty hard to maintain across languages. The more code we can scaffold the better our ability to port drivers is. This will definitely put a tremendous pressure on core to keep it consistent (read/write buffers but also conversation contexts). Before we go there it would be great to define major principles for architectural parts we have defined in core. Best, Łukasz On 20.04.2021 18:29, Christofer Dutz wrote: > Just cause I looked it up in order to tease Sebastian: > > https://github.com/apache/plc4x/tree/milestone/last-daffodil/protocols > /s7/src/main/resources/org/apache/plc4x/protocols/s7 > > That is generally what we're talking about ... the protocol.dfdl.xsd was the > definition of the datatypes and how to serialize and parse them (The syntax) > ... so an equivalent to our mspec. The scxml.xml file is the semantik .. it > has the message templates and how they interact. > > With this I was able to interact with my real S7-1200 with a driver > that was almost fully specified (Problem then was ... it's insanely > slow and SCXML wouldn't allow concurrent execution of things ... and I > think it's still not released) > > Chris > > > -----Ursprüngliche Nachricht----- > Von: Christofer Dutz <[email protected]> > Gesendet: Dienstag, 20. April 2021 18:21 > An: [email protected] > Betreff: AW: Next evolution of code generation > > You forgot, > > [Stage 6] Buy Siemens and fill the headquater with ball bath balls and > have all the Managers serve cotton candy to Kids that get to play > there for free all day :-) > > But back to being serious :-) > > In java it's not that bad to assemble the messages that the drivers send, in > Go it however sucks quite a bit and in C it's a nightmare. So I would love to > automate that process. > > I had a lengthy chat with Sebastian today. He was showing me his new XML > serialization. I then remembered how I brought up the idea of a > MessageFactory as a next logical step some time ago and initially thought > about using the test-data in the Integration-tests as a basis. Unfortunately > generating code from that would have been quite a task, because the Jackson > XML format was missing quite some information. Sebastian's new format however > has all of this information and it would therefore be quite simple to create > a code-generator from this. > > Therefore, it would be great, if we would simply define a list of pre-defined > messages in an XML document that we put alongside the mspec. > > Right now, I would probably simply fire an XSLT on the XML ... this would > mean that for every language we would also create an XSLT for the > MessageFactory. > > We could then either use normal Maven plugins, or - what I would prefer - > give our code-generator a second mojo that we would simply call to generate > this. The configuration needed when using default maven plugins would > probably scare people and if we hide this complexity in our plugin, nobody > will complain. > > I would assume, that we don't need to define 3 additional formats. I think > something like I built with daffodil and this strange xml statemachine > project, should also work: > - Mspec for syntax > - XML for semantic > > So, I'm in strong favor of starting this journey. > > Chris > > -----Ursprüngliche Nachricht----- > Von: Sebastian Rühl <[email protected]> > Gesendet: Dienstag, 20. April 2021 17:58 > An: [email protected] > Betreff: Next evolution of code generation > > Hi Together, > > Having worked now on golang for a while and jumped back to java I saw that we > had many similarities between the languages. First of all the structure if > message being sent (e.g. during the S7 connection dance) and the > conversations happening (e.g. during the s7 connect). So it feels more and > more like and extension to our current code generation: > > [Stage 1] At the moment we are in Stage 1: > - We can generate Typed structure into the languages from mspec (The > definition of what is contained in an instance of a message) > - We can generate „how“ the message is serialized on to its primary > medium In byte form (xml, string, etc will be decoupled soon from > these models, as they currently re-implement the [de]serialization in > a different format [first fruits from that already in develop]) > - We can generate some known constants from mspec > > At this point we have at least a reproducible serialization between > languages (assuming the templates are correctly implemented). [For > that I also have began changing the way xml (un)marshaling works which > will replace the current one soon which I will talk about more in a > later mail] > > [Stage 2] We can generate Message instances in a template like code > - also a sister spec > - Taking S7 for example: during a connection initalization you need to > send 3 specific defined messages (with potential little variation) to > the plc. These messages need to be re-implemented in every language > the same way > - Defining a sister-spec beside *.mspec we could define these message with > placeholders and generate a MessageTypeFactory from this to create known > messages. > - Another example is the ADS symbolic addressing where you send a well > defined message and get the response for the index offset in the response. > > [Stage 3] We can generate an interaction for a ping-pong > - also a sister spec > - e.g. Sending request x with parameters x1, x2 we expect response > with value z1,z2 > - defining this we could define a simple request/response and generate > code from that > > [Stage 4] We can generate an conversation > - also a sister spec > - going back to the s7 connection dance, we could possible using the > features from the stages before define a whole well defined flow: > „Send COTP Connection Request->Receive COTP Connection Response->Send > S7 Connection Request->Receive S7 Connection Response->Send S7 > Identification Request->Receive S7 Identification Response“ > - at the moment we do something like this already in our driver test suites > but we don’t generate code from that. > > [Stage 5] We can define properties and options > - at the moment we need to reimplement necessary options and parameters in > every language again. e.g. AmsNetId, S7-Rack and so one. > - defining known and required options we could generate configuration > parsers in every language > > [Stage 6] World domination > - We ran out of stages and there is nothing more left to do for plc4x > :o) > > So now as the wall of text has ended I would be glad to get some feedback and > ideas on the above points. > > Disclaimer: points above came up while having a fruitful OTR > discussion with Chris. So credit where credit is due :) > > Sebastian >
