Chris,
I suggest that since you are clearly familiar with binary data and network packets, PCAP is a good schema for you to look at w.r.t. how DFDL schemas work, so as to learn from an example. Web search for "DFDLSchemas PCAP" to find it. ...mike beckerle ________________________________ From: Christofer Dutz <christofer.d...@c-ware.de> Sent: Thursday, January 10, 2019 11:01:58 AM To: dev@daffodil.apache.org Subject: Re: Using DFDL to generate model, parser and generator? Perhaps I should start with a simpler protocol, that simply wraps binary content in a small packet with a small binary header ... Should start with "The Works (TM)" ... Outlook for Android<https://aka.ms/ghei36> herunterladen ________________________________ From: Christofer Dutz Sent: Thursday, January 10, 2019 4:49:53 PM To: dev@daffodil.apache.org Subject: Re: Using DFDL to generate model, parser and generator? (This time the full message) Hi Mike, so I converted one of my Protocols into a Xml-Schema with some utilization of the DFDL namespace (Trying to get started) Unfortunately I'm having a little problem with how to define type inheritance ... so I have for example parameter elements which all start with a one byte type-code followed by a one byte length parameter. The rest is completely different, based on the type of parameter. Seems something like this isn't DFDL: <xs:complexType name="S7Message"> <xs:sequence> <!-- S7 Magic Byte always 0x32 --> <xs:element name="magicByte" type="xs:unsignedByte" fixed="50"/> <xs:element name="messageType" type="xs:unsignedByte"/> <!-- Reserved value always 0x0000 --> <xs:element name="reserved" type="xs:unsignedShort" fixed="0"/> <xs:element name="tpduReference" type="xs:unsignedShort"/> <xs:element name="parametersLength" type="xs:unsignedShort"/> <xs:element name="payloadsLength" type="xs:unsignedShort"/> </xs:sequence> </xs:complexType> <xs:complexType name="S7RequestMessage"> <xs:complexContent> <xs:extension base="s7:S7Message"> <xs:sequence> <xs:element name="parameters" type="s7:S7RequestParameter" minOccurs="0" maxOccurs="unbounded"/> <xs:element name="payloads" type="s7:S7RequestPayload" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType> <xs:complexType name="S7ResponseMessage"> <xs:complexContent> <xs:extension base="s7:S7Message"> <xs:sequence> <xs:element name="errorClass" type="xs:unsignedByte"/> <xs:element name="errorCode" type="xs:unsignedByte"/> <xs:element name="parameters" type="s7:S7ResponseParameter" minOccurs="0" maxOccurs="unbounded"/> <xs:element name="payloads" type="s7:S7ResponsePayload" minOccurs="0" maxOccurs="unbounded"/> </xs:sequence> </xs:extension> </xs:complexContent> </xs:complexType> In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of it to do it's job, is that correct? I thought at first that if it's an extension I could start with a schema and have a look as what it does and then to iteratively narrow it down, but it seems that approach isn't valied. Think first I need to learn how to do, what I want in DFDL. But I did encounter some things that might be problematic (perhaps) So sometimes I read a byte that contains a number of elements or a length of an element and have to then read exactly this number of bytes or exactly this number of parameters which summed up size matches a total parameter size ... Hope it is possible to model stuff like this with DFDL. Chris [1] https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java Am 10.01.19, 14:47 schrieb "Beckerle, Mike" <mbecke...@tresys.com>: This make sense to me architecturally as infrastructure means by which people use this. Compiling a DFDL schema into a any sort of compiled form, whether that is generated code, or just a saved runtime data structure (like we have now) is exactly what people want as a maven/sbt build step, so creating a plugin that does this is very sensible. Right now compiling is slow (unnecessarily. I hope we speed it up soon, and reduce it's memory footprint), so a build step that is only re-run if the schema actually changed is very useful to save time waiting around for the Daffodil compiler. I suggest that the generation of code from the daffodil parser/unparser data structures will push the boundaries of what anyone would call "template". This is going to be a quite sophisticated recursive descent walk, accumulating a variety of things and eventually emitting the code. I think it is totally worth it to try this though. ________________________________ From: Christofer Dutz <christofer.d...@c-ware.de> Sent: Thursday, January 10, 2019 4:57:22 AM To: dev@daffodil.apache.org Subject: Re: Using DFDL to generate model, parser and generator? Hi Mike, Well I am currently experimenting with creating a DFDL schema for one of the many protocol layers we have. I would propose the following (Please correct me, if I'm wrong): - We create DFDL Schemas - We use Daffodil to process these (Assuming that in order to process DFDL schemas, there has to be some sort of model representation) - We add a Maven plugin, that uses the parsed schema representation model and allows generating code via some templating language (Freemarker and Velocity are both Apache ... so should be one of these) - In a project you define templates for the current usecase (A general purpose runtime would be sub-optimal for our case ... we would probably use Netty utils for parsing/serializing) Perhaps based on these PLC4X templates it would make sense to build other sets of templates as part of the Daffodil project. Daffodil could have multiple sets of templates for different languages and frameworks. Eventually a template module could have a runtime module to be used in the code generated. So you would use the maven plugin without providing a template-artifact and it would look for local templates. If however you provide a template-artifact, then the plugin would use those. In the end I would probably build the maven plugin in a way that it makes things easier to run it on the Command line or build plugins for SBT, Gradle, Ant whatsoever ... What do you think? Chris Am 09.01.19, 20:10 schrieb "Beckerle, Mike" <mbecke...@tresys.com>: Christofer, Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly this right now. Effectively what you are describing is a code-generator backend for Daffodil. I think this is a great idea. I personally want to have one that generates VHDL or Verilog or other Hardware synthesis language so you can go direct to an FPGA for data parsing at hardware speed. Anyway, such a generator would likely be adding to the existing parser/unparser primitives so that in addition to having parse() and unparse() methods, they would have generateCode() methods that emit the equivalent code, and recursively invoke the sub-objects to generateCode() that is incorporated recursively. I would suggest that the existing Daffodil backend, which may well not be fast enough for your needs, would nevertheless be very valuable part of your testing strategy as your schemas should work on Daffodil, and you can then verify that the parser behavior from your generated code is consistent. It also may be helpful for diagnostic purposes - ie., if data is parsed and determined invalid, perhaps your "kit" to help your users involves parsing such data with regular old Daffodil into XML for tangibility/inspection. There is a fair amount of runtime-library to be created to go with the generated code of course. Daffodil has daffodil-lib, daffdil-io, daffodil-runtime1, and daffodil-runtime1-unparser, each of which contains a large volume of runtime code that would need to be replaced with C/C++ equivalent in a new runtime. I would suggest much of the work is actually here, not in the compilation. I really hope you undertake this effort. I think it will be a big value-add to Daffodil if it has a code-gen style backend. The current back-end really hasn't had raw-speed as its goal. It has largely been about correctness, and getting the DFDL standard fully/mostly implemented quickly. Let us know how we can help you get started. The other thing worth mentioning is that Daffodil does have on roadmap, plans to create a streaming parser/unparser. This would not build a DOM-tree like structure, but would instead emit events along the lines of a SAX-style parse of data. Now some formats are simply not stream-able, and there is no option to avoid building up a tree in memory. But many formats are stream-able, and people really do want the ability to parse files much larger than memory, in finite RAM, so long as the format is streamable. -mike beckerle Tresys Technology ________________________________ From: Christofer Dutz <christofer.d...@c-ware.de> Sent: Wednesday, January 9, 2019 8:56:28 AM To: dev@daffodil.apache.org Subject: Using DFDL to generate model, parser and generator? Hi all, I am currently looking for a solution to the following question: In the Apache PLC4X (incubating) project we are implementing a lot of different industry protocols. Each protocol sends packets following a particular format. For each of these we currently implement an internal model, serializers and parsers. Till now this has been pure Java, but we are now starting to work on C++ and would like to add even more languages. As we don’t want to manually keep in sync all of these implementations, my idea was to describe the data format in some form and have the parsers, serializers and the model generated from that. So the implementation only has to take care of the plumbing and the state-machine of the protocol. In Montreal I attended a great talk on DFDL and Daffodil, so I think DFDL in general would be a great fit. Unfortunately we don’t want to parse any data format into an XML or DOM representation for performance reasons. My ideal workflow would look like this: 1. For every protocol I define the DFDL documents describing the different types of messages for a given protocol 2. I define multiple protocol implementation modules (one for each language) 3. I use a maven plugin in each of these to generate the code for that particular language from those central DFDL definitions Is this possible? Is it planned to support this in the future? What other options do you see for this sort of problem? I am absolutely willing to get my hands dirty and help implement this, if you say: “Yes we want that too but haven’t managed to do that yet”. Chris
