Hi Sebastian, thanks for this summary and thanks for your hard work on all of that in the last few weeks. Looking forward to profiting from these changes (Currently I'm stuck with other things in my working time and in my private time I'm working hard on getting my I40 factory working)
Chris -----Ursprüngliche Nachricht----- Von: Sebastian Rühl <[email protected]> Gesendet: Montag, 26. April 2021 11:46 An: [email protected] Betreff: plc4x summary from 1bd132a to f97d36f (buffer changes and test framework) TLDR: changes in plc4x which help developing drivers across languages that have no effect how plc4x send/receiver data or how you use the API As of the last mail `Progress on plc4go` from 15.03.2021 I want to give another overview what has happened between 1bd132a to f97d36f Changes on read/write buffers In plc4go and plc4j Note the below changes don’t have any impact on the API on plc4x nor on the way data is written onto the wire Existing buffers (which were byte base [writing and reading bytes to a underlying reader/writer]) had an interface introduced which abstracted the interaction from plc4x from the representation (reason for that described in point 5). Write buffer 2 New methods added PushContext: Indicates that the model is about to write data which is enclosed by a logical context PopContext: Indicates that the model is done writing data which is enclosed by a logical context Every method got a logicalName and a varargs/variadic writerArguments added LogicalName: indicates the logical name of the data which is about to be written (e.g. checksum flag, address) WriterArguments: are optional hints about the data being written (e.g. name of the enum [e.g.ADS_READ] being written, or if a logical context is a list) The above points are completely ignored by the byte based writers (which got renamed with the introduction of the interface to *ByteBased) as the above changes don’t have any impact on byte based writing (They could be used for log tracing for better debugging in the future) Read buffer 2 New methods added PullContext: indicates that the model is about to read data which is enclosed by a logical context CloseContext: Indicates that the model is done reading data which is enclosed by a logical context Every method got a logicalName and a varargs/variadic readerArguments added LogicalName: indicates the logical name of the data which is about to be read (e.g. checksum flag, address) ReaderArguments: are optional hints about the data being read (e.g. name of the enum [e.g.ADS_READ] being written, or if a logical context is a list) The above points are completely ignored by the byte based readers (which got renamed with the introduction of the interface to *ByteBased) as the above changes don’t have any impact on byte based writing (They could be used for log tracing for better debugging in the future) On plc4j the old methods are still available as default methods but will be deprecated in the future (The buffers are meant to be used in the context of the generated models which uses the new interfaces) In the future I plan on adding new byte/byte[] based methods to better encapsulate logical context on byte/byte[] fields which can then be better rendered by implementors of the buffer interfaces (mostly required by non byte based buffers). See my other mail on the mailing list about bytes <https://lists.apache.org/thread.html/r5285838fd0b53d50fca34c2539b2d12ac275361d4faf5df36f3ec804@%3Cdev.plc4x.apache.org%3E>. Refactoring in golang SPI Added default implementations for connection and codec Moved default implementation into a spi/default package These can be configured with variadic WithOption calls Moved duplicated codes into commons classes At the moment these a realized with a cycling dependency for callback which get refactored to a non cyclic approach in the future Added default implementations for request and response Removed cyclic dependency from request interceptor Ported the request transaction manager from java At the moment this is a 1:1 port from java and doesn’t make full use of the native golang features. The TM was required by the s7 driver to prevent clients from overloading a plc4x (which could lead to a huge RL crash ;) ) Added Ascii/Xml/Json buffers described in point 4 and 5 Updated goDocs S7 Driver in plc4go Implemented the S7 driver into plc4go In the course of this I fixed several bugs and initially with the ads drivers lead to many changes described in this mail Added a s7Io_test.go (will be helpful to visualize <https://github.com/apache/plc4x/blob/develop/plc4go/internal/plc4go/s7/s7Io_test.go> many changes described in this mail) AsciiBox/HexWriter in plc4go During development of ads and s7 driver I had several smaller bugs (e.g. padding issues which got fixed) were I had to stare at bytes every day till I almost could identify <https://github.com/apache/plc4x/blob/develop/plc4go/internal/plc4go/s7/s7Io_test.go#L349> a s7Message by looking at the hex at the end of the day 🤪 This lead me to writing a small tool which would help me to visually segment the message into logical context (see Point 1) to better look for flipping bytes. The AsciiBox was born which takes a string and draws a nice box <https://github.com/apache/plc4x/blob/develop/plc4go/internal/plc4go/spi/utils/asciiBox_test.go#L79> (requires a boxed font which apparently GitHub doesn’t have; on boxed fonts these are square 😉) around it (with a optional name) Implementing this I added a `Box() utils.AsciiBox` to every model which renders itself and Childs to a fixed width box (120.) which could overflow a bit if required In the initial version which doesn’t use the new buffers (see point 4) a nice hex box could be drawn for bytes arrays. Requiring to render hexdumps for data segments into a dynamic box required a dynamic width hex dump. So I introduced hex.go which could resize to any width. (Other than normal hex dumps this tool uses a base 10 byte index [normal use hex dumpers and render 16 bytes each line] which made working with them easier that hex indexes [most of us count in base 10 anyway]) During later development a BufferBoxedBased was introduced to render the messages true to their byte representation. New xml/json writers in plc4go and plc4j Important Background info: Implementing the BoxWriter (see point 4.7) I noticed that I implemented the serializing/deserializing of message twice (duplicated code that does the same but different is never good). I noticed that we already had the code for serializing/deserializing messages to/from the wire and it might be better to reuse this code for that. This initially this lead to the introduction of the interfaces described in Point 1 (This also explains why I need a logical name and context for asciibox which will also get now relevant for json xml). I also noticed that the xml was imitating the byte serializing/deserializing (in golang). Even worse: Between java and golang there were differences (Java used annotations and Jackson for rendering / golang used a second generated implementation). This made it hard when handling language agnostic xml test suites. I introduced xml/json based writers which implemented the buffer interfaces and deterministically (at least the should but at the moment they need a bit improvement [e.g. plc4go json writing]) write/read xml/json (xml has a fixed element oder json not which requires buffering on json at least when reading). The model itself dictates when to read and to write and the writing/reading/rendering is decoupled and encapsulated in the concrete read/writer implementation (json/xml/ascii[go only]). Whenever possible existing code was changed to use the new xml readers/writers (e.g. point 6) Updated xml representation for test suites (Parser/Driver) Having now the new buffer interfaces and their xml read/writer I could change <https://github.com/apache/plc4x/commit/8f28d7965555ed1823425e565e5a4f5972f378ef#diff-5579ab196820eb2cdccdd7c7a429b35823e80aacd493038a1c0f13a3c025dce9R503> the test suites. The xml contains now more information (at points a bit too much because there is no support for byte/byte[] rendering at the moment see point 1.6). Remaining java class names from test read request will be removed in future changes and replaced by language agnostic representation Updated xsd for suites to include protocol name and flavor (which is required to reliable identify message types in golang and java) Xml now contains additional attributes describing the basic datatype and bit length (these are optional which are not required for parsing as the buffer dictates the type on a read operation anyway but helps us humans to get more context about the datatype. At the moment these get still validated by the read buffers anyway to check for consistency [which will be configurable if not already]) Feature update in golang api Builders are now real chain able builders Updated examples and documentation for the new chain able builders Change is non breaking as the return type was changed from void to builder Added pl4go config package to configure tracing of workers and transactions Updated most golang docs in the api Several Bugfixes (see commit messages in Ref compare at the bottom) Refactored Testsuites to break down the big class in java and add support for the buffers described in point 5 Much more There are more changes contained and if you want to keep reading for whatever reason (maybe you like reading long mails 😁) you can check the commit descriptions (which are mostly quite verbose) in the GitHub/compare <https://github.com/apache/plc4x/compare/1bd132a241ef7f8e6603450514f9d970c8dae91e...f97d36f51b8942b2a4e9f03328ae364edd73aa5d>. Also let me know in this tread if I overlooked a important change which isn’t self describing or there are required features missing which don’t have a todo in the code. Thats for that and thanks for reading this far 😀 Sebastian Ps: (Ref compare GitHub/compare <https://github.com/apache/plc4x/compare/1bd132a241ef7f8e6603450514f9d970c8dae91e...f97d36f51b8942b2a4e9f03328ae364edd73aa5d>)
