Re: Using DFDL to generate model, parser and generator?
into a Xml-Schema with some > utilization of the DFDL namespace (Trying to get started) > > Unfortunately I'm having a little problem with how to define type > inheritance ... so I have for example parameter elements which all start with > a one byte type-code followed by a one byte length parameter. > > The rest is completely different, based on the type of parameter. > > > > Seems something like this isn't DFDL: > > > > > > > > > > fixed="50"/> > > > > > > fixed="0"/> > > > > type="xs:unsignedShort"/> > > > > > > > > > > > > > > > > > > type="s7:S7RequestParameter" minOccurs="0" maxOccurs="unbounded"/> > > type="s7:S7RequestPayload" minOccurs="0" maxOccurs="unbounded"/> > > > > > > > > > > > > > > > > > > > > type="xs:unsignedByte"/> > > type="xs:unsignedByte"/> > > type="s7:S7ResponseParameter" minOccurs="0" maxOccurs="unbounded"/> > > type="s7:S7ResponsePayload" minOccurs="0" maxOccurs="unbounded"/> > > > > > > > > > > > > In the end it seems that DFDL doesn't extend Xml Schema, but uses a > subset of it to do it's job, is that correct? > > > > I thought at first that if it's an extension I could start with a > schema and have a look as what it does and then to iteratively narrow it > down, but it seems that approach isn't valied. > > > > Think first I need to learn how to do, what I want in DFDL. But I did > encounter some things that might be problematic (perhaps) > > > > So sometimes I read a byte that contains a number of elements or a > length of an element and have to then read exactly this number of bytes or > exactly this number of parameters which summed up size matches a total > parameter size ... > > Hope it is possible to model stuff like this with DFDL. > > > > Chris > > > > > > > > > > [1] > https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java > > > > > > Am 10.01.19, 14:47 schrieb "Beckerle, Mike" : > > > > This make sense to me architecturally as infrastructure means by > which people use this. > > > > > > Compiling a DFDL schema into a any sort of compiled form, whether > that is generated code, or just a saved runtime data structure (like we have > now) is exactly what people want as a maven/sbt build step, so creating a > plugin that does this is very sensible. > > > > > > Right now compiling is slow (unnecessarily. I hope we speed it up > soon, and reduce it's memory footprint), so a build step that is only re-run > if the schema actually changed is very useful to save time waiting around for > the Daffodil compiler. > > > > > > I suggest that the generation of code from the daffodil > parser/unparser data structures will push the boundaries of what anyone would > call "template". This is going to be a quite sophisticated recursive descent > walk, accumulating a variety of things and eventually emitting the code. I > think it is totally worth it to try this though. > > > > > > From: Christofer Dutz > > Sent: Thursday, January 10, 2019 4:57:22 AM > > To: dev@daffodil.apache.org > > Subject: Re: Using DFDL to generate model, parser and generator? > > > > Hi Mike, > > > > Well I am currently experimenting with creating a DFDL schema for > one of the many protocol layers we have. > > > > I would propose the following (Please correct me, if I'm wron
Re: Using DFDL to generate model, parser and generator?
> > Seems something like this isn't DFDL: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of it to do it's job, is that correct? > > I thought at first that if it's an extension I could start with a schema and have a look as what it does and then to iteratively narrow it down, but it seems that approach isn't valied. > > Think first I need to learn how to do, what I want in DFDL. But I did encounter some things that might be problematic (perhaps) > > So sometimes I read a byte that contains a number of elements or a length of an element and have to then read exactly this number of bytes or exactly this number of parameters which summed up size matches a total parameter size ... > Hope it is possible to model stuff like this with DFDL. > > Chris > > > > > [1] https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java > > > Am 10.01.19, 14:47 schrieb "Beckerle, Mike" : > > This make sense to me architecturally as infrastructure means by which people use this. > > > Compiling a DFDL schema into a any sort of compiled form, whether that is generated code, or just a saved runtime data structure (like we have now) is exactly what people want as a maven/sbt build step, so creating a plugin that does this is very sensible. > > > Right now compiling is slow (unnecessarily. I hope we speed it up soon, and reduce it's memory footprint), so a build step that is only re-run if the schema actually changed is very useful to save time waiting around for the Daffodil compiler. > > > I suggest that the generation of code from the daffodil parser/unparser data structures will push the boundaries of what anyone would call "template". This is going to be a quite sophisticated recursive descent walk, accumulating a variety of things and eventually emitting the code. I think it is totally worth it to try this though. > > > From: Christofer Dutz > Sent: Thursday, January 10, 2019 4:57:22 AM > To: dev@daffodil.apache.org > Subject: Re: Using DFDL to generate model, parser and generator? > > Hi Mike, > > Well I am currently experimenting with creating a DFDL schema for one of the many protocol layers we have. > > I would propose the following (Please correct me, if I'm wrong): > - We create DFDL Schemas > - We use Daffodil to process these (Assuming that in order to process DFDL schemas, there has to be some sort of model representation) > - We add a Maven plugin, that uses the parsed schema representation model and allows generating code via some templating language (Freemarker and Velocity are both Apache ... so should be one of these) > - In a project you define templates for the current usecase (A general purpose runtime would be sub-optimal for our case ... we would probably use Netty utils for parsing/serializing) > > Perhaps based on these PLC4X templates it would make sense to build other sets of templates as part of the Daffodil project. > Daffodil could have multiple sets of templates for different languages and frameworks. Eventually a template module could have a runtime module to be used in the code generated. > > So you would use the maven plugin without providing a template-artifact and it would look for local templates. If however you provide a template-artifact, then the plugin would use those. > > In the end I would probably build the maven plugin in a way that it makes things easier to run it on the Command line or build plugins for SBT, Gradle, Ant whatsoever ... > > What do you think? > > Chris > > > > Am 09.01.19, 20:10 schrieb "Beckerle, Mike" : > > Christofer, > > > Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly
Re: Using DFDL to generate model, parser and generator?
Chris,
I suggest that since you are clearly familiar with binary data and network
packets, PCAP is a good schema for you to look at w.r.t. how DFDL schemas work,
so as to learn from an example.
Web search for "DFDLSchemas PCAP" to find it.
...mike beckerle
From: Christofer Dutz
Sent: Thursday, January 10, 2019 11:01:58 AM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
Perhaps I should start with a simpler protocol, that simply wraps binary
content in a small packet with a small binary header ... Should start with "The
Works (TM)" ...
Outlook for Android<https://aka.ms/ghei36> herunterladen
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:49:53 PM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
(This time the full message)
Hi Mike,
so I converted one of my Protocols into a Xml-Schema with some utilization of
the DFDL namespace (Trying to get started)
Unfortunately I'm having a little problem with how to define type inheritance
... so I have for example parameter elements which all start with a one byte
type-code followed by a one byte length parameter.
The rest is completely different, based on the type of parameter.
Seems something like this isn't DFDL:
In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of
it to do it's job, is that correct?
I thought at first that if it's an extension I could start with a schema and
have a look as what it does and then to iteratively narrow it down, but it
seems that approach isn't valied.
Think first I need to learn how to do, what I want in DFDL. But I did encounter
some things that might be problematic (perhaps)
So sometimes I read a byte that contains a number of elements or a length of an
element and have to then read exactly this number of bytes or exactly this
number of parameters which summed up size matches a total parameter size ...
Hope it is possible to model stuff like this with DFDL.
Chris
[1]
https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java
Am 10.01.19, 14:47 schrieb "Beckerle, Mike" :
This make sense to me architecturally as infrastructure means by which
people use this.
Compiling a DFDL schema into a any sort of compiled form, whether that is
generated code, or just a saved runtime data structure (like we have now) is
exactly what people want as a maven/sbt build step, so creating a plugin that
does this is very sensible.
Right now compiling is slow (unnecessarily. I hope we speed it up soon, and
reduce it's memory footprint), so a build step that is only re-run if the
schema actually changed is very useful to save time waiting around for the
Daffodil compiler.
I suggest that the generation of code from the daffodil parser/unparser
data structures will push the boundaries of what anyone would call "template".
This is going to be a quite sophisticated recursive descent walk, accumulating
a variety of things and eventually emitting the code. I think it is totally
worth it to try this though.
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:57:22 AM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
Hi Mike,
Well I am currently experimenting with creating a DFDL schema for one of
the many protocol layers we have.
I would propose the following (Please correct me, if I'm wrong):
- We create DFDL Schemas
- We use Daffodil to process these (Assuming that in order to process DFDL
schemas, there has to be some sort of model representation)
- We add a Maven plugin, that uses the parsed schema representation model
and allows generating code via some templating language (Freemarker and
Velocity are both Apache ... so should be one of these)
- In a project you define templates for the current usecase (A general
purpose runtime would be sub-optimal for our case ... we would probably use
Netty utils for parsing/serializing)
Perhaps based on these PLC4X templates it would make sense to build other
sets of templates as part of the Daffodil project.
Daffodil could have multiple sets of templates for different languages and
frameworks. Eventually a template module could have a runtime module to be used
in the code generated.
So you would use
Re: Using DFDL to generate model, parser and generator?
Hi Chris, As you've found out, DFDL only allows for a limited subset of XML schema, and inheritance is not one of those features it allows. Usually you can accomplish the same thing via custom types and groups. For example, you could change the S7Message from a complexType to a group, and then reference that group in the Request/Response elements, e.g.: Regarding your last question, DFDL handles parse time determine length/occurances/etc. using DFDL expressions, which are a subset of XPath. For example, if you have a dynmic lenth, you might have something like this: So first a 4 byte length is parsed, and then a hexBinary blob of data is parsed where the length is determined by the expression that gets the value of the parsed length value. For variable occurences, it might look something like this: So in this case, we parse a 4 byte int for the number of occurrences. At runtime, we determine the value of the parsed occurs element and have than many repeats of the 4-byte payloads element. The XPath expression language is complex it enough that it should allow to perform whatever math might be necessary to calculate sums of sizes and the like. Section 23 of the DFDL spec [1] describes the expression language in more detail. Section 23.4 defines a grammar for subset of XPath that is supported. - Steve [1] https://daffodil.apache.org/docs/dfdl/#_Toc398030820 On 1/10/19 10:49 AM, Christofer Dutz wrote: > (This time the full message) > > Hi Mike, > > so I converted one of my Protocols into a Xml-Schema with some utilization of > the DFDL namespace (Trying to get started) > Unfortunately I'm having a little problem with how to define type inheritance > ... so I have for example parameter elements which all start with a one byte > type-code followed by a one byte length parameter. > The rest is completely different, based on the type of parameter. > > Seems something like this isn't DFDL: > > > > > > > > > > > > > > > > > > > type="s7:S7RequestParameter" minOccurs="0" maxOccurs="unbounded"/> > minOccurs="0" maxOccurs="unbounded"/> > > > > > > > > > > > > type="s7:S7ResponseParameter" minOccurs="0" maxOccurs="unbounded"/> > minOccurs="0" maxOccurs="unbounded"/> > > > > > > In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of > it to do it's job, is that correct? > > I thought at first that if it's an extension I could start with a schema and > have a look as what it does and then to iteratively narrow it down, but it > seems that approach isn't valied. > > Think first I need to learn how to do, what I want in DFDL. But I did > encounter some things that might be problematic (perhaps) > > So sometimes I read a byte that contains a number of elements or a length of > an element and have to then read exactly this number of bytes or exactly this > number of parameters which summed up size matches a total parameter size ... > Hope it is possible to model stuff like this with DFDL. > > Chris > > > > > [1] > https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java > > > Am 10.01.19, 14:47 schrieb "Beckerle, Mike" : > > This make sense to me architecturally as infrastructure means by which > people use this. > > > Compiling a DFDL schema into a any sort of compiled form, whether that is > generated code, or just a saved runtime data structure (like we have now) is > exactly what people want as a maven/sbt build step, so creating a plugin that > does this is very sensible. > > > Right now compiling is slow (unnecessarily. I hope we speed it up soon, > and reduce it's memory footprint), so a build step that is only re-run if the > schema actually changed is very useful to save time waiting around for the > Daffodil compiler. > > > I suggest that the generation of code from the daffodil parser/unparser > data structures will push the boundaries of what anyone would call > "template". This is going to be a quite sophisticated recur
Re: Using DFDL to generate model, parser and generator?
Perhaps I should start with a simpler protocol, that simply wraps binary
content in a small packet with a small binary header ... Should start with "The
Works (TM)" ...
Outlook for Android<https://aka.ms/ghei36> herunterladen
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:49:53 PM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
(This time the full message)
Hi Mike,
so I converted one of my Protocols into a Xml-Schema with some utilization of
the DFDL namespace (Trying to get started)
Unfortunately I'm having a little problem with how to define type inheritance
... so I have for example parameter elements which all start with a one byte
type-code followed by a one byte length parameter.
The rest is completely different, based on the type of parameter.
Seems something like this isn't DFDL:
In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of
it to do it's job, is that correct?
I thought at first that if it's an extension I could start with a schema and
have a look as what it does and then to iteratively narrow it down, but it
seems that approach isn't valied.
Think first I need to learn how to do, what I want in DFDL. But I did encounter
some things that might be problematic (perhaps)
So sometimes I read a byte that contains a number of elements or a length of an
element and have to then read exactly this number of bytes or exactly this
number of parameters which summed up size matches a total parameter size ...
Hope it is possible to model stuff like this with DFDL.
Chris
[1]
https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java
Am 10.01.19, 14:47 schrieb "Beckerle, Mike" :
This make sense to me architecturally as infrastructure means by which
people use this.
Compiling a DFDL schema into a any sort of compiled form, whether that is
generated code, or just a saved runtime data structure (like we have now) is
exactly what people want as a maven/sbt build step, so creating a plugin that
does this is very sensible.
Right now compiling is slow (unnecessarily. I hope we speed it up soon, and
reduce it's memory footprint), so a build step that is only re-run if the
schema actually changed is very useful to save time waiting around for the
Daffodil compiler.
I suggest that the generation of code from the daffodil parser/unparser
data structures will push the boundaries of what anyone would call "template".
This is going to be a quite sophisticated recursive descent walk, accumulating
a variety of things and eventually emitting the code. I think it is totally
worth it to try this though.
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:57:22 AM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
Hi Mike,
Well I am currently experimenting with creating a DFDL schema for one of
the many protocol layers we have.
I would propose the following (Please correct me, if I'm wrong):
- We create DFDL Schemas
- We use Daffodil to process these (Assuming that in order to process DFDL
schemas, there has to be some sort of model representation)
- We add a Maven plugin, that uses the parsed schema representation model
and allows generating code via some templating language (Freemarker and
Velocity are both Apache ... so should be one of these)
- In a project you define templates for the current usecase (A general
purpose runtime would be sub-optimal for our case ... we would probably use
Netty utils for parsing/serializing)
Perhaps based on these PLC4X templates it would make sense to build other
sets of templates as part of the Daffodil project.
Daffodil could have multiple sets of templates for different languages and
frameworks. Eventually a template module could have a runtime module to be used
in the code generated.
So you would use the maven plugin without providing a template-artifact and
it would look for local templates. If however you provide a template-artifact,
then the plugin would use those.
In the end I would probably build the maven plugin in a way that it makes
things easier to run it on the Command line or build plugins for SBT, Gradle,
Ant whatsoever ...
What do you think?
Chris
Am 09.01.19, 20:10 schrieb "Beckerle, Mike" :
Christof
Re: Using DFDL to generate model, parser and generator?
(This time the full message)
Hi Mike,
so I converted one of my Protocols into a Xml-Schema with some utilization of
the DFDL namespace (Trying to get started)
Unfortunately I'm having a little problem with how to define type inheritance
... so I have for example parameter elements which all start with a one byte
type-code followed by a one byte length parameter.
The rest is completely different, based on the type of parameter.
Seems something like this isn't DFDL:
In the end it seems that DFDL doesn't extend Xml Schema, but uses a subset of
it to do it's job, is that correct?
I thought at first that if it's an extension I could start with a schema and
have a look as what it does and then to iteratively narrow it down, but it
seems that approach isn't valied.
Think first I need to learn how to do, what I want in DFDL. But I did encounter
some things that might be problematic (perhaps)
So sometimes I read a byte that contains a number of elements or a length of an
element and have to then read exactly this number of bytes or exactly this
number of parameters which summed up size matches a total parameter size ...
Hope it is possible to model stuff like this with DFDL.
Chris
[1]
https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java
Am 10.01.19, 14:47 schrieb "Beckerle, Mike" :
This make sense to me architecturally as infrastructure means by which
people use this.
Compiling a DFDL schema into a any sort of compiled form, whether that is
generated code, or just a saved runtime data structure (like we have now) is
exactly what people want as a maven/sbt build step, so creating a plugin that
does this is very sensible.
Right now compiling is slow (unnecessarily. I hope we speed it up soon, and
reduce it's memory footprint), so a build step that is only re-run if the
schema actually changed is very useful to save time waiting around for the
Daffodil compiler.
I suggest that the generation of code from the daffodil parser/unparser
data structures will push the boundaries of what anyone would call "template".
This is going to be a quite sophisticated recursive descent walk, accumulating
a variety of things and eventually emitting the code. I think it is totally
worth it to try this though.
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:57:22 AM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
Hi Mike,
Well I am currently experimenting with creating a DFDL schema for one of
the many protocol layers we have.
I would propose the following (Please correct me, if I'm wrong):
- We create DFDL Schemas
- We use Daffodil to process these (Assuming that in order to process DFDL
schemas, there has to be some sort of model representation)
- We add a Maven plugin, that uses the parsed schema representation model
and allows generating code via some templating language (Freemarker and
Velocity are both Apache ... so should be one of these)
- In a project you define templates for the current usecase (A general
purpose runtime would be sub-optimal for our case ... we would probably use
Netty utils for parsing/serializing)
Perhaps based on these PLC4X templates it would make sense to build other
sets of templates as part of the Daffodil project.
Daffodil could have multiple sets of templates for different languages and
frameworks. Eventually a template module could have a runtime module to be used
in the code generated.
So you would use the maven plugin without providing a template-artifact and
it would look for local templates. If however you provide a template-artifact,
then the plugin would use those.
In the end I would probably build the maven plugin in a way that it makes
things easier to run it on the Command line or build plugins for SBT, Gradle,
Ant whatsoever ...
What do you think?
Chris
Am 09.01.19, 20:10 schrieb "Beckerle, Mike" :
Christofer,
Yes what you suggest is possible, is what many people want, has been
talked about here and there, but I don't know of anyone else doing exactly this
right now.
Effectively what you are describing is a code-generator backend for
Daffodil. I think this is a great idea. I personally want to have one that
generates VHD
Re: Using DFDL to generate model, parser and generator?
Hi Mike,
so I converted one of my Protocols into a Xml-Schema with some utilization of
the DFDL namespace (Trying to get started)
Unfortunately I'm having a little problem with how to define type inheritance
... so I have for example parameter elements which all start with a one byte
type-code followed by a one byte length parameter.
The rest is completely different, based on the type of parameter.
Seems something like this isn't DFDL:
[1]
https://github.com/OpenDFDL/examples/blob/master/helloWorld/src/main/java/HelloWorld.java
Am 10.01.19, 14:47 schrieb "Beckerle, Mike" :
This make sense to me architecturally as infrastructure means by which
people use this.
Compiling a DFDL schema into a any sort of compiled form, whether that is
generated code, or just a saved runtime data structure (like we have now) is
exactly what people want as a maven/sbt build step, so creating a plugin that
does this is very sensible.
Right now compiling is slow (unnecessarily. I hope we speed it up soon, and
reduce it's memory footprint), so a build step that is only re-run if the
schema actually changed is very useful to save time waiting around for the
Daffodil compiler.
I suggest that the generation of code from the daffodil parser/unparser
data structures will push the boundaries of what anyone would call "template".
This is going to be a quite sophisticated recursive descent walk, accumulating
a variety of things and eventually emitting the code. I think it is totally
worth it to try this though.
From: Christofer Dutz
Sent: Thursday, January 10, 2019 4:57:22 AM
To: dev@daffodil.apache.org
Subject: Re: Using DFDL to generate model, parser and generator?
Hi Mike,
Well I am currently experimenting with creating a DFDL schema for one of
the many protocol layers we have.
I would propose the following (Please correct me, if I'm wrong):
- We create DFDL Schemas
- We use Daffodil to process these (Assuming that in order to process DFDL
schemas, there has to be some sort of model representation)
- We add a Maven plugin, that uses the parsed schema representation model
and allows generating code via some templating language (Freemarker and
Velocity are both Apache ... so should be one of these)
- In a project you define templates for the current usecase (A general
purpose runtime would be sub-optimal for our case ... we would probably use
Netty utils for parsing/serializing)
Perhaps based on these PLC4X templates it would make sense to build other
sets of templates as part of the Daffodil project.
Daffodil could have multiple sets of templates for different languages and
frameworks. Eventually a template module could have a runtime module to be used
in the code generated.
So you would use the maven plugin without providing a template-artifact and
it would look for local templates. If however you provide a template-artifact,
then the plugin would use those.
In the end I would probably build the maven plugin in a way that it makes
things easier to run it on the Command line or build plugins for SBT, Gradle,
Ant whatsoever ...
What do you think?
Chris
Am 09.01.19, 20:10 schrieb "Beckerle, Mike" :
Christofer,
Yes what you suggest is possible, is what many people want, has been
talked about here and there, but I don't know of anyone else doing exactly this
right now.
Effectively what you are describing is a code-generator backend for
Daffodil. I think this is a great idea. I personally want to have one that
generates VHDL or Verilog or other Hardware synthesis language so you can go
direct to an FPGA for data parsing at hardware speed.
Anyway, such a generator would likely be adding to the existing
parser/unparser primitives so that in addition to having parse() and unparse()
methods, they would have generateCode() methods that emit the equivalent code,
and recursively invoke the sub-objects to generateCode() that is incorporated
recursively.
I would suggest that the existing Daffodil backend, which may well not
be fast enough for your needs, would nevertheless be very valuable part of your
testing strategy as your schemas should work on Daffodil, and you can then
verify that the parser behavior from your generated code is consistent. It
also may be helpful for diagnostic purposes - ie., if data is parsed and
determined invalid, perhaps your "kit" to help your users involves parsing such
data with regular old Daffodil into XML for tangibility/inspection.
There is a fair amount of runtime-library to be created to go with the
generated code of course. Daffodil has daffodil-lib, daffdil-io,
daffodil-runtime1, and daffodil-r
Re: Using DFDL to generate model, parser and generator?
This make sense to me architecturally as infrastructure means by which people use this. Compiling a DFDL schema into a any sort of compiled form, whether that is generated code, or just a saved runtime data structure (like we have now) is exactly what people want as a maven/sbt build step, so creating a plugin that does this is very sensible. Right now compiling is slow (unnecessarily. I hope we speed it up soon, and reduce it's memory footprint), so a build step that is only re-run if the schema actually changed is very useful to save time waiting around for the Daffodil compiler. I suggest that the generation of code from the daffodil parser/unparser data structures will push the boundaries of what anyone would call "template". This is going to be a quite sophisticated recursive descent walk, accumulating a variety of things and eventually emitting the code. I think it is totally worth it to try this though. From: Christofer Dutz Sent: Thursday, January 10, 2019 4:57:22 AM To: dev@daffodil.apache.org Subject: Re: Using DFDL to generate model, parser and generator? Hi Mike, Well I am currently experimenting with creating a DFDL schema for one of the many protocol layers we have. I would propose the following (Please correct me, if I'm wrong): - We create DFDL Schemas - We use Daffodil to process these (Assuming that in order to process DFDL schemas, there has to be some sort of model representation) - We add a Maven plugin, that uses the parsed schema representation model and allows generating code via some templating language (Freemarker and Velocity are both Apache ... so should be one of these) - In a project you define templates for the current usecase (A general purpose runtime would be sub-optimal for our case ... we would probably use Netty utils for parsing/serializing) Perhaps based on these PLC4X templates it would make sense to build other sets of templates as part of the Daffodil project. Daffodil could have multiple sets of templates for different languages and frameworks. Eventually a template module could have a runtime module to be used in the code generated. So you would use the maven plugin without providing a template-artifact and it would look for local templates. If however you provide a template-artifact, then the plugin would use those. In the end I would probably build the maven plugin in a way that it makes things easier to run it on the Command line or build plugins for SBT, Gradle, Ant whatsoever ... What do you think? Chris Am 09.01.19, 20:10 schrieb "Beckerle, Mike" : Christofer, Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly this right now. Effectively what you are describing is a code-generator backend for Daffodil. I think this is a great idea. I personally want to have one that generates VHDL or Verilog or other Hardware synthesis language so you can go direct to an FPGA for data parsing at hardware speed. Anyway, such a generator would likely be adding to the existing parser/unparser primitives so that in addition to having parse() and unparse() methods, they would have generateCode() methods that emit the equivalent code, and recursively invoke the sub-objects to generateCode() that is incorporated recursively. I would suggest that the existing Daffodil backend, which may well not be fast enough for your needs, would nevertheless be very valuable part of your testing strategy as your schemas should work on Daffodil, and you can then verify that the parser behavior from your generated code is consistent. It also may be helpful for diagnostic purposes - ie., if data is parsed and determined invalid, perhaps your "kit" to help your users involves parsing such data with regular old Daffodil into XML for tangibility/inspection. There is a fair amount of runtime-library to be created to go with the generated code of course. Daffodil has daffodil-lib, daffdil-io, daffodil-runtime1, and daffodil-runtime1-unparser, each of which contains a large volume of runtime code that would need to be replaced with C/C++ equivalent in a new runtime. I would suggest much of the work is actually here, not in the compilation. I really hope you undertake this effort. I think it will be a big value-add to Daffodil if it has a code-gen style backend. The current back-end really hasn't had raw-speed as its goal. It has largely been about correctness, and getting the DFDL standard fully/mostly implemented quickly. Let us know how we can help you get started. The other thing worth mentioning is that Daffodil does have on roadmap, plans to create a streaming parser/unparser. This would not build a DOM-tree like structure, but would instead emit events along the lines of a SAX-style parse of data. Now some form
Re: Using DFDL to generate model, parser and generator?
Hi Mike, Well I am currently experimenting with creating a DFDL schema for one of the many protocol layers we have. I would propose the following (Please correct me, if I'm wrong): - We create DFDL Schemas - We use Daffodil to process these (Assuming that in order to process DFDL schemas, there has to be some sort of model representation) - We add a Maven plugin, that uses the parsed schema representation model and allows generating code via some templating language (Freemarker and Velocity are both Apache ... so should be one of these) - In a project you define templates for the current usecase (A general purpose runtime would be sub-optimal for our case ... we would probably use Netty utils for parsing/serializing) Perhaps based on these PLC4X templates it would make sense to build other sets of templates as part of the Daffodil project. Daffodil could have multiple sets of templates for different languages and frameworks. Eventually a template module could have a runtime module to be used in the code generated. So you would use the maven plugin without providing a template-artifact and it would look for local templates. If however you provide a template-artifact, then the plugin would use those. In the end I would probably build the maven plugin in a way that it makes things easier to run it on the Command line or build plugins for SBT, Gradle, Ant whatsoever ... What do you think? Chris Am 09.01.19, 20:10 schrieb "Beckerle, Mike" : Christofer, Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly this right now. Effectively what you are describing is a code-generator backend for Daffodil. I think this is a great idea. I personally want to have one that generates VHDL or Verilog or other Hardware synthesis language so you can go direct to an FPGA for data parsing at hardware speed. Anyway, such a generator would likely be adding to the existing parser/unparser primitives so that in addition to having parse() and unparse() methods, they would have generateCode() methods that emit the equivalent code, and recursively invoke the sub-objects to generateCode() that is incorporated recursively. I would suggest that the existing Daffodil backend, which may well not be fast enough for your needs, would nevertheless be very valuable part of your testing strategy as your schemas should work on Daffodil, and you can then verify that the parser behavior from your generated code is consistent. It also may be helpful for diagnostic purposes - ie., if data is parsed and determined invalid, perhaps your "kit" to help your users involves parsing such data with regular old Daffodil into XML for tangibility/inspection. There is a fair amount of runtime-library to be created to go with the generated code of course. Daffodil has daffodil-lib, daffdil-io, daffodil-runtime1, and daffodil-runtime1-unparser, each of which contains a large volume of runtime code that would need to be replaced with C/C++ equivalent in a new runtime. I would suggest much of the work is actually here, not in the compilation. I really hope you undertake this effort. I think it will be a big value-add to Daffodil if it has a code-gen style backend. The current back-end really hasn't had raw-speed as its goal. It has largely been about correctness, and getting the DFDL standard fully/mostly implemented quickly. Let us know how we can help you get started. The other thing worth mentioning is that Daffodil does have on roadmap, plans to create a streaming parser/unparser. This would not build a DOM-tree like structure, but would instead emit events along the lines of a SAX-style parse of data. Now some formats are simply not stream-able, and there is no option to avoid building up a tree in memory. But many formats are stream-able, and people really do want the ability to parse files much larger than memory, in finite RAM, so long as the format is streamable. -mike beckerle Tresys Technology From: Christofer Dutz Sent: Wednesday, January 9, 2019 8:56:28 AM To: dev@daffodil.apache.org Subject: Using DFDL to generate model, parser and generator? Hi all, I am currently looking for a solution to the following question: In the Apache PLC4X (incubating) project we are implementing a lot of different industry protocols. Each protocol sends packets following a particular format. For each of these we currently implement an internal model, serializers and parsers. Till now this has been pure Java, but we are now starting to work on C++ and would like to add even more languages. As we don’t want to manually keep in sync all of these implementations, my idea was to describe the data format in
Re: Using DFDL to generate model, parser and generator?
Christofer, Yes what you suggest is possible, is what many people want, has been talked about here and there, but I don't know of anyone else doing exactly this right now. Effectively what you are describing is a code-generator backend for Daffodil. I think this is a great idea. I personally want to have one that generates VHDL or Verilog or other Hardware synthesis language so you can go direct to an FPGA for data parsing at hardware speed. Anyway, such a generator would likely be adding to the existing parser/unparser primitives so that in addition to having parse() and unparse() methods, they would have generateCode() methods that emit the equivalent code, and recursively invoke the sub-objects to generateCode() that is incorporated recursively. I would suggest that the existing Daffodil backend, which may well not be fast enough for your needs, would nevertheless be very valuable part of your testing strategy as your schemas should work on Daffodil, and you can then verify that the parser behavior from your generated code is consistent. It also may be helpful for diagnostic purposes - ie., if data is parsed and determined invalid, perhaps your "kit" to help your users involves parsing such data with regular old Daffodil into XML for tangibility/inspection. There is a fair amount of runtime-library to be created to go with the generated code of course. Daffodil has daffodil-lib, daffdil-io, daffodil-runtime1, and daffodil-runtime1-unparser, each of which contains a large volume of runtime code that would need to be replaced with C/C++ equivalent in a new runtime. I would suggest much of the work is actually here, not in the compilation. I really hope you undertake this effort. I think it will be a big value-add to Daffodil if it has a code-gen style backend. The current back-end really hasn't had raw-speed as its goal. It has largely been about correctness, and getting the DFDL standard fully/mostly implemented quickly. Let us know how we can help you get started. The other thing worth mentioning is that Daffodil does have on roadmap, plans to create a streaming parser/unparser. This would not build a DOM-tree like structure, but would instead emit events along the lines of a SAX-style parse of data. Now some formats are simply not stream-able, and there is no option to avoid building up a tree in memory. But many formats are stream-able, and people really do want the ability to parse files much larger than memory, in finite RAM, so long as the format is streamable. -mike beckerle Tresys Technology From: Christofer Dutz Sent: Wednesday, January 9, 2019 8:56:28 AM To: dev@daffodil.apache.org Subject: Using DFDL to generate model, parser and generator? Hi all, I am currently looking for a solution to the following question: In the Apache PLC4X (incubating) project we are implementing a lot of different industry protocols. Each protocol sends packets following a particular format. For each of these we currently implement an internal model, serializers and parsers. Till now this has been pure Java, but we are now starting to work on C++ and would like to add even more languages. As we don’t want to manually keep in sync all of these implementations, my idea was to describe the data format in some form and have the parsers, serializers and the model generated from that. So the implementation only has to take care of the plumbing and the state-machine of the protocol. In Montreal I attended a great talk on DFDL and Daffodil, so I think DFDL in general would be a great fit. Unfortunately we don’t want to parse any data format into an XML or DOM representation for performance reasons. My ideal workflow would look like this: 1. For every protocol I define the DFDL documents describing the different types of messages for a given protocol 2. I define multiple protocol implementation modules (one for each language) 3. I use a maven plugin in each of these to generate the code for that particular language from those central DFDL definitions Is this possible? Is it planned to support this in the future? What other options do you see for this sort of problem? I am absolutely willing to get my hands dirty and help implement this, if you say: “Yes we want that too but haven’t managed to do that yet”. Chris
