Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi all,

I’ve just put up a blog-entry explaining the rationale behind yaml and device 
tree.

https://www.konsulko.com/yaml-and-device-tree/

Please give it a quick glance and give feedback if you can.

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi all,

I’ve just put up a blog-entry explaining the rationale behind yaml and device 
tree.

https://www.konsulko.com/yaml-and-device-tree/

Please give it a quick glance and give feedback if you can.

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Sun, Oct 22, 2017 at 7:23 PM, Pantelis Antoniou
 wrote:
> Hi Grant,
>
>> On Oct 22, 2017, at 19:54 , Grant Likely  wrote:
>>
>> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Grant,
>>>
 On Oct 20, 2017, at 20:46 , Grant Likely  wrote:

 On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
  wrote:
[...]
>>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>>> the crazy things that are possible (and actually used in practice) with DTS.
>>
>> I don’t think it is as dire as that. The DTS structure is not complex
>> and I think can easily be mapped into pure YAML. But, it does require
>> treating the DTS structure separately from its semantic meaning. For
>> example, the grammar of nodes, properties and labels easily maps to
>> pure YAML, but phandles and overlay trees have semantic meaning that
>> must be resolved by DT specific code. I’ll respond to you’re specific
>> examples below…
>>
>
> We are in complete agreement here. Single nodes and properties map to YAML 
> perfectly.
> It’s the complex way that we build the complete DTB that stress the YAML 
> structures.

Okay, good.

>>> If we were to force YAML/DT file to be regular YAML files parseable by 
>>> everything
>>> we’d have to rethink a lot of conventions that DT files currently use and 
>>> I’m afraid
>>> a machine translation from DTS to YAML might not be feasible then.
>>>
>>> OTOH moving to that model might make it possible to use YAML constructs 
>>> that are not
>>> mapped at all to DTS. For instance
>>>
>>> - foo: true
>>>  bar: “string”
>>>
>>> - foo: false
>>>  bar: “another-string”
>>>
>>> is not possible to be mapped to a DT node/property structure right now.
>>
>> I’m not following. Are you saying DT has no way to encode a list of
>> nodes? What use case are you imagining?
>>
>
> This threw me off too at first.
>
> The problem is when you try to convert this into a live tree structure.
>
> The YAML encoding is
>
> +SEQ
>  +MAP
>   =VAL :foo
>   =VAL :true
>   =VAL :bar
>   =VAL "string
>  -MAP
>  +MAP
>   =VAL :foo
>   =VAL :false
>   =VAL :bar
>   =VAL "another-string
>  -MAP
> -SEQ
>
> So it’s a sequence of MAPs.
>
> In a live tree DTS form would be
>
> noname-property = { foo=true; bar = “string”; }, { foo = false; 
> bar=“another-string”; };
>
> We don’t have the capability as right now to express something like this.
>
> Namely properties containing nodes, and the root of the live tree not being a 
> node.

Ah, so you're proposing property data that encodes key/value pairs,
correct? Effectively adding another layer of namespace under a
property, but that namespace is invisible to and generic DT parsing
code in an OS (for how DT is defined today).

I would straight out make that structure illegal in the YAML DT
metaschema until such time as the semantics are defined. For now
properties should only be allowed to contain scalars, or lists of
scalars/lists which can be nested.

[...]

 The encoding differentiates between nodes and properties implicitly
 base on whether the contents are a map, or a scalar/list. This does
 mean any parser needs to do a bit more work and it may impact what can
 be done with validation (I'm not going to talk about validation in
 this email though. We'll talk next week.)
>>>
>>> Hmm, I’m not sure I follow this. Is that related to the metadata problem.
>>
>> It has to do with iterating over properties or nodes. Nodes and props
>> are mixed together, and any code processing them has to explicitly
>> check the type to differentiate between the two. The alternative would
>> be to collect all properties under a single key:
>>
>> parent:
>>  $props:
>>foo: hello
>>bar: world
>>  child1: {}
>>  child2: {}
>>
>
> That’s no problem. It does look weird though. I am using the same parser
> for parsing bindings and this might make the files a little bit weird.

That it does. I don’t feel strongly about either approach, but I want
to see how the different encodings affect doing the schema bits.

>
> Keep in mind that we have two YAML formats to parse; the hardware description
> and the bindings describing them.

Three! Hw description, schema & metaschema. :-) But metaschema should
be a normal jsonschema file.

>
>>>
>>> BTW, a blog post explaining the rationale behind yamldt is going to come
>>> up soon at our website, I’ll post the link when it does.
>>
>> Cool. I see it got posted, but didn’t get a chance to read it before I
>> got on this airplane. Will try to read this evening.
>>
>> (BTW, I’m arriving at about 11:00 tonight)
>>
>
> Have a safe flight, you’ll get to beat me up tomorrow :)

See you tomorrow. It will be good to catch up.

g.

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Sun, Oct 22, 2017 at 7:23 PM, Pantelis Antoniou
 wrote:
> Hi Grant,
>
>> On Oct 22, 2017, at 19:54 , Grant Likely  wrote:
>>
>> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Grant,
>>>
 On Oct 20, 2017, at 20:46 , Grant Likely  wrote:

 On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
  wrote:
[...]
>>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>>> the crazy things that are possible (and actually used in practice) with DTS.
>>
>> I don’t think it is as dire as that. The DTS structure is not complex
>> and I think can easily be mapped into pure YAML. But, it does require
>> treating the DTS structure separately from its semantic meaning. For
>> example, the grammar of nodes, properties and labels easily maps to
>> pure YAML, but phandles and overlay trees have semantic meaning that
>> must be resolved by DT specific code. I’ll respond to you’re specific
>> examples below…
>>
>
> We are in complete agreement here. Single nodes and properties map to YAML 
> perfectly.
> It’s the complex way that we build the complete DTB that stress the YAML 
> structures.

Okay, good.

>>> If we were to force YAML/DT file to be regular YAML files parseable by 
>>> everything
>>> we’d have to rethink a lot of conventions that DT files currently use and 
>>> I’m afraid
>>> a machine translation from DTS to YAML might not be feasible then.
>>>
>>> OTOH moving to that model might make it possible to use YAML constructs 
>>> that are not
>>> mapped at all to DTS. For instance
>>>
>>> - foo: true
>>>  bar: “string”
>>>
>>> - foo: false
>>>  bar: “another-string”
>>>
>>> is not possible to be mapped to a DT node/property structure right now.
>>
>> I’m not following. Are you saying DT has no way to encode a list of
>> nodes? What use case are you imagining?
>>
>
> This threw me off too at first.
>
> The problem is when you try to convert this into a live tree structure.
>
> The YAML encoding is
>
> +SEQ
>  +MAP
>   =VAL :foo
>   =VAL :true
>   =VAL :bar
>   =VAL "string
>  -MAP
>  +MAP
>   =VAL :foo
>   =VAL :false
>   =VAL :bar
>   =VAL "another-string
>  -MAP
> -SEQ
>
> So it’s a sequence of MAPs.
>
> In a live tree DTS form would be
>
> noname-property = { foo=true; bar = “string”; }, { foo = false; 
> bar=“another-string”; };
>
> We don’t have the capability as right now to express something like this.
>
> Namely properties containing nodes, and the root of the live tree not being a 
> node.

Ah, so you're proposing property data that encodes key/value pairs,
correct? Effectively adding another layer of namespace under a
property, but that namespace is invisible to and generic DT parsing
code in an OS (for how DT is defined today).

I would straight out make that structure illegal in the YAML DT
metaschema until such time as the semantics are defined. For now
properties should only be allowed to contain scalars, or lists of
scalars/lists which can be nested.

[...]

 The encoding differentiates between nodes and properties implicitly
 base on whether the contents are a map, or a scalar/list. This does
 mean any parser needs to do a bit more work and it may impact what can
 be done with validation (I'm not going to talk about validation in
 this email though. We'll talk next week.)
>>>
>>> Hmm, I’m not sure I follow this. Is that related to the metadata problem.
>>
>> It has to do with iterating over properties or nodes. Nodes and props
>> are mixed together, and any code processing them has to explicitly
>> check the type to differentiate between the two. The alternative would
>> be to collect all properties under a single key:
>>
>> parent:
>>  $props:
>>foo: hello
>>bar: world
>>  child1: {}
>>  child2: {}
>>
>
> That’s no problem. It does look weird though. I am using the same parser
> for parsing bindings and this might make the files a little bit weird.

That it does. I don’t feel strongly about either approach, but I want
to see how the different encodings affect doing the schema bits.

>
> Keep in mind that we have two YAML formats to parse; the hardware description
> and the bindings describing them.

Three! Hw description, schema & metaschema. :-) But metaschema should
be a normal jsonschema file.

>
>>>
>>> BTW, a blog post explaining the rationale behind yamldt is going to come
>>> up soon at our website, I’ll post the link when it does.
>>
>> Cool. I see it got posted, but didn’t get a chance to read it before I
>> got on this airplane. Will try to read this evening.
>>
>> (BTW, I’m arriving at about 11:00 tonight)
>>
>
> Have a safe flight, you’ll get to beat me up tomorrow :)

See you tomorrow. It will be good to catch up.

g.

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 22, 2017, at 19:54 , Grant Likely  wrote:
> 
> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
>  wrote:
>> Hi Grant,
>> 
>>> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
>>> 
>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>>  wrote:
 Hello again,
 
 Significant progress has been made on yamldt and is now capable of
 not only generating yaml from DTS source but also compiling DTS sources
 and being almost fully compatible with DTC.
 
 Compiling the kernel's DTBs using yamldt is as simple as using a
 DTC=yamldt.
 
 Error reporting is accurate and validation against a YAML based schema
 works as well. In a short while I will begin posting patches with
 fixes on bindings and DTS files in the kernel.
 
 Please try it on your platform and report if you encounter any problems.
 
 https://github.com/pantoniou/yamldt
 
 I am eagerly awaiting for your comments.
>>> 
>>> Hi Pantelis,
>>> 
>>> This is good work. I've played around with it and I'm looking forward
>>> to chatting next week.
>>> 
>> 
>> Thanks. I’m looking forward to it too.
>> 
>>> One thing I've done is tried loading the output YAML files into
>>> another YAML interpreter and the current encoding causes problems.
>>> Specifically, in yamldt anchors/aliases are being used as a
>>> replacement for labels/phandles, but that conflicts with the YAML data
>>> model which defines a reference as a way to make a copy of the data
>>> appear in another part of the tree. For example, for the following
>>> snippit:
>>> 
>>> intc: intc@1 {
>>>   #interrupt-cells = <1>;
>>>   compatible = "acme,intc";
>>>   reg = <0x1 0x1000>;
>>>   gpio-controller;
>>> };
>>> 
>>> serial@2 {
>>>   compatible = "acme,uart";
>>>   reg = <0x2 0x1000>;
>>>   interrupt-parent = <>;
>>>   interrupts = <5>;
>>> };
>>> 
>>> yamldt will encode this as:
>>> 
>>> intc@1: 
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@2:
>>>   compatible: acme,uart
>>>   reg: [0x2, 0x1000]
>>>   interrupt-parent: *intc
>>>   interrupts: 5
>>> 
>>> But, the expected behaviour for a YAML parser is expand the alias
>>> '*intc' which results in the following structure:
>>> 
>>> intc@1: 
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@2:
>>>   compatible: acme,uart
>>>   reg: [0x2, 0x1000]
>>>   interrupt-parent:
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>>   interrupts: 5
>>> 
>>> See? It results in the entire interrupt controller node to appear as
>>> an instance under the interrupt-parent property, when the intention is
>>> only to create a phandle. yamldt should not redefine the behaviour of
>>> '*' aliases. Instead, it should use a different indicator, either
>>> using an explicit !phandle tag, or by replacing '*' with something
>>> else. I worked around it in my tests by replacing '*' with '$’.
>> 
>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>> the crazy things that are possible (and actually used in practice) with DTS.
> 
> I don’t think it is as dire as that. The DTS structure is not complex
> and I think can easily be mapped into pure YAML. But, it does require
> treating the DTS structure separately from its semantic meaning. For
> example, the grammar of nodes, properties and labels easily maps to
> pure YAML, but phandles and overlay trees have semantic meaning that
> must be resolved by DT specific code. I’ll respond to you’re specific
> examples below…
> 

We are in complete agreement here. Single nodes and properties map to YAML 
perfectly.
It’s the complex way that we build the complete DTB that stress the YAML 
structures.

>> 
>> For instance there’s no way a normal YAML parser won’t choke with something 
>> like
>> 
>> / {
>>  foo;
>> };
>> 
>> / {
>>  bar;
>> };
>> 
>> Which is a common idiom in DTS files.
> 
> That’s only true when the top level of nodes is encoded as a map, but
> it shouldn’t be. It should be a list instead, for two reasons. First,
> order matters for the top level. Subsequent top level trees have to be
> applied in order to fully resolve the tree, but a map doesn’t preserve
> the ordering. Second, as you rightly point out, the same name can be
> used for multiple top level trees. So, the encoding should be a list
> with each list entry containing the node path {/path,}, and the
> node data. One possible way to do this is with a tuple:
> 
> treedata:
> - - /
>  - foo:
> - - /
>  - bar:
> 
> Another option is to add a special property in the node to contain the name:
> 
> treedata:
> - $path: /
>  foo:
> - $path: /
>  bar:
> 
> Personally, I prefer 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 22, 2017, at 19:54 , Grant Likely  wrote:
> 
> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
>  wrote:
>> Hi Grant,
>> 
>>> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
>>> 
>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>>  wrote:
 Hello again,
 
 Significant progress has been made on yamldt and is now capable of
 not only generating yaml from DTS source but also compiling DTS sources
 and being almost fully compatible with DTC.
 
 Compiling the kernel's DTBs using yamldt is as simple as using a
 DTC=yamldt.
 
 Error reporting is accurate and validation against a YAML based schema
 works as well. In a short while I will begin posting patches with
 fixes on bindings and DTS files in the kernel.
 
 Please try it on your platform and report if you encounter any problems.
 
 https://github.com/pantoniou/yamldt
 
 I am eagerly awaiting for your comments.
>>> 
>>> Hi Pantelis,
>>> 
>>> This is good work. I've played around with it and I'm looking forward
>>> to chatting next week.
>>> 
>> 
>> Thanks. I’m looking forward to it too.
>> 
>>> One thing I've done is tried loading the output YAML files into
>>> another YAML interpreter and the current encoding causes problems.
>>> Specifically, in yamldt anchors/aliases are being used as a
>>> replacement for labels/phandles, but that conflicts with the YAML data
>>> model which defines a reference as a way to make a copy of the data
>>> appear in another part of the tree. For example, for the following
>>> snippit:
>>> 
>>> intc: intc@1 {
>>>   #interrupt-cells = <1>;
>>>   compatible = "acme,intc";
>>>   reg = <0x1 0x1000>;
>>>   gpio-controller;
>>> };
>>> 
>>> serial@2 {
>>>   compatible = "acme,uart";
>>>   reg = <0x2 0x1000>;
>>>   interrupt-parent = <>;
>>>   interrupts = <5>;
>>> };
>>> 
>>> yamldt will encode this as:
>>> 
>>> intc@1: 
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@2:
>>>   compatible: acme,uart
>>>   reg: [0x2, 0x1000]
>>>   interrupt-parent: *intc
>>>   interrupts: 5
>>> 
>>> But, the expected behaviour for a YAML parser is expand the alias
>>> '*intc' which results in the following structure:
>>> 
>>> intc@1: 
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>> 
>>> serial@2:
>>>   compatible: acme,uart
>>>   reg: [0x2, 0x1000]
>>>   interrupt-parent:
>>>   "#interrupt-cells": 1
>>>   compatible: acme,intc
>>>   reg: [0x1, 0x1000]
>>>   gpio-controller:
>>>   interrupts: 5
>>> 
>>> See? It results in the entire interrupt controller node to appear as
>>> an instance under the interrupt-parent property, when the intention is
>>> only to create a phandle. yamldt should not redefine the behaviour of
>>> '*' aliases. Instead, it should use a different indicator, either
>>> using an explicit !phandle tag, or by replacing '*' with something
>>> else. I worked around it in my tests by replacing '*' with '$’.
>> 
>> Yes. This is expected. I don’t think pure YAML form is a good match for all
>> the crazy things that are possible (and actually used in practice) with DTS.
> 
> I don’t think it is as dire as that. The DTS structure is not complex
> and I think can easily be mapped into pure YAML. But, it does require
> treating the DTS structure separately from its semantic meaning. For
> example, the grammar of nodes, properties and labels easily maps to
> pure YAML, but phandles and overlay trees have semantic meaning that
> must be resolved by DT specific code. I’ll respond to you’re specific
> examples below…
> 

We are in complete agreement here. Single nodes and properties map to YAML 
perfectly.
It’s the complex way that we build the complete DTB that stress the YAML 
structures.

>> 
>> For instance there’s no way a normal YAML parser won’t choke with something 
>> like
>> 
>> / {
>>  foo;
>> };
>> 
>> / {
>>  bar;
>> };
>> 
>> Which is a common idiom in DTS files.
> 
> That’s only true when the top level of nodes is encoded as a map, but
> it shouldn’t be. It should be a list instead, for two reasons. First,
> order matters for the top level. Subsequent top level trees have to be
> applied in order to fully resolve the tree, but a map doesn’t preserve
> the ordering. Second, as you rightly point out, the same name can be
> used for multiple top level trees. So, the encoding should be a list
> with each list entry containing the node path {/path,}, and the
> node data. One possible way to do this is with a tuple:
> 
> treedata:
> - - /
>  - foo:
> - - /
>  - bar:
> 
> Another option is to add a special property in the node to contain the name:
> 
> treedata:
> - $path: /
>  foo:
> - $path: /
>  bar:
> 
> Personally, I prefer the special property approach. That would also be
> a good way to encode labels
> 

The YAML bare sequence is 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
 wrote:
> Hi Grant,
>
>> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
>>
>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>  wrote:
>>> Hello again,
>>>
>>> Significant progress has been made on yamldt and is now capable of
>>> not only generating yaml from DTS source but also compiling DTS sources
>>> and being almost fully compatible with DTC.
>>>
>>> Compiling the kernel's DTBs using yamldt is as simple as using a
>>> DTC=yamldt.
>>>
>>> Error reporting is accurate and validation against a YAML based schema
>>> works as well. In a short while I will begin posting patches with
>>> fixes on bindings and DTS files in the kernel.
>>>
>>> Please try it on your platform and report if you encounter any problems.
>>>
>>> https://github.com/pantoniou/yamldt
>>>
>>> I am eagerly awaiting for your comments.
>>
>> Hi Pantelis,
>>
>> This is good work. I've played around with it and I'm looking forward
>> to chatting next week.
>>
>
> Thanks. I’m looking forward to it too.
>
>> One thing I've done is tried loading the output YAML files into
>> another YAML interpreter and the current encoding causes problems.
>> Specifically, in yamldt anchors/aliases are being used as a
>> replacement for labels/phandles, but that conflicts with the YAML data
>> model which defines a reference as a way to make a copy of the data
>> appear in another part of the tree. For example, for the following
>> snippit:
>>
>> intc: intc@1 {
>>#interrupt-cells = <1>;
>>compatible = "acme,intc";
>>reg = <0x1 0x1000>;
>>gpio-controller;
>> };
>>
>> serial@2 {
>>compatible = "acme,uart";
>>reg = <0x2 0x1000>;
>>interrupt-parent = <>;
>>interrupts = <5>;
>> };
>>
>> yamldt will encode this as:
>>
>> intc@1: 
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>
>> serial@2:
>>compatible: acme,uart
>>reg: [0x2, 0x1000]
>>interrupt-parent: *intc
>>interrupts: 5
>>
>> But, the expected behaviour for a YAML parser is expand the alias
>> '*intc' which results in the following structure:
>>
>> intc@1: 
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>
>> serial@2:
>>compatible: acme,uart
>>reg: [0x2, 0x1000]
>>interrupt-parent:
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>interrupts: 5
>>
>> See? It results in the entire interrupt controller node to appear as
>> an instance under the interrupt-parent property, when the intention is
>> only to create a phandle. yamldt should not redefine the behaviour of
>> '*' aliases. Instead, it should use a different indicator, either
>> using an explicit !phandle tag, or by replacing '*' with something
>> else. I worked around it in my tests by replacing '*' with '$’.
>
> Yes. This is expected. I don’t think pure YAML form is a good match for all
> the crazy things that are possible (and actually used in practice) with DTS.

I don’t think it is as dire as that. The DTS structure is not complex
and I think can easily be mapped into pure YAML. But, it does require
treating the DTS structure separately from its semantic meaning. For
example, the grammar of nodes, properties and labels easily maps to
pure YAML, but phandles and overlay trees have semantic meaning that
must be resolved by DT specific code. I’ll respond to you’re specific
examples below...

>
> For instance there’s no way a normal YAML parser won’t choke with something 
> like
>
> / {
>   foo;
> };
>
> / {
>   bar;
> };
>
> Which is a common idiom in DTS files.

That’s only true when the top level of nodes is encoded as a map, but
it shouldn’t be. It should be a list instead, for two reasons. First,
order matters for the top level. Subsequent top level trees have to be
applied in order to fully resolve the tree, but a map doesn’t preserve
the ordering. Second, as you rightly point out, the same name can be
used for multiple top level trees. So, the encoding should be a list
with each list entry containing the node path {/path,}, and the
node data. One possible way to do this is with a tuple:

treedata:
- - /
  - foo:
- - /
  - bar:

Another option is to add a special property in the node to contain the name:

treedata:
- $path: /
  foo:
- $path: /
  bar:

Personally, I prefer the special property approach. That would also be
a good way to encode labels

> The decision to use the YAML anchors and references was more borne out of a 
> desire
> to sort of match conceptually the labels and references of DTS. It’s not a big
> issue to switch to something different.

I think it would be a good idea. One reason for defining a YAML
encoding is to be able to use existing tools to work with the data.
Changing the 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou
 wrote:
> Hi Grant,
>
>> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
>>
>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>>  wrote:
>>> Hello again,
>>>
>>> Significant progress has been made on yamldt and is now capable of
>>> not only generating yaml from DTS source but also compiling DTS sources
>>> and being almost fully compatible with DTC.
>>>
>>> Compiling the kernel's DTBs using yamldt is as simple as using a
>>> DTC=yamldt.
>>>
>>> Error reporting is accurate and validation against a YAML based schema
>>> works as well. In a short while I will begin posting patches with
>>> fixes on bindings and DTS files in the kernel.
>>>
>>> Please try it on your platform and report if you encounter any problems.
>>>
>>> https://github.com/pantoniou/yamldt
>>>
>>> I am eagerly awaiting for your comments.
>>
>> Hi Pantelis,
>>
>> This is good work. I've played around with it and I'm looking forward
>> to chatting next week.
>>
>
> Thanks. I’m looking forward to it too.
>
>> One thing I've done is tried loading the output YAML files into
>> another YAML interpreter and the current encoding causes problems.
>> Specifically, in yamldt anchors/aliases are being used as a
>> replacement for labels/phandles, but that conflicts with the YAML data
>> model which defines a reference as a way to make a copy of the data
>> appear in another part of the tree. For example, for the following
>> snippit:
>>
>> intc: intc@1 {
>>#interrupt-cells = <1>;
>>compatible = "acme,intc";
>>reg = <0x1 0x1000>;
>>gpio-controller;
>> };
>>
>> serial@2 {
>>compatible = "acme,uart";
>>reg = <0x2 0x1000>;
>>interrupt-parent = <>;
>>interrupts = <5>;
>> };
>>
>> yamldt will encode this as:
>>
>> intc@1: 
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>
>> serial@2:
>>compatible: acme,uart
>>reg: [0x2, 0x1000]
>>interrupt-parent: *intc
>>interrupts: 5
>>
>> But, the expected behaviour for a YAML parser is expand the alias
>> '*intc' which results in the following structure:
>>
>> intc@1: 
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>
>> serial@2:
>>compatible: acme,uart
>>reg: [0x2, 0x1000]
>>interrupt-parent:
>>"#interrupt-cells": 1
>>compatible: acme,intc
>>reg: [0x1, 0x1000]
>>gpio-controller:
>>interrupts: 5
>>
>> See? It results in the entire interrupt controller node to appear as
>> an instance under the interrupt-parent property, when the intention is
>> only to create a phandle. yamldt should not redefine the behaviour of
>> '*' aliases. Instead, it should use a different indicator, either
>> using an explicit !phandle tag, or by replacing '*' with something
>> else. I worked around it in my tests by replacing '*' with '$’.
>
> Yes. This is expected. I don’t think pure YAML form is a good match for all
> the crazy things that are possible (and actually used in practice) with DTS.

I don’t think it is as dire as that. The DTS structure is not complex
and I think can easily be mapped into pure YAML. But, it does require
treating the DTS structure separately from its semantic meaning. For
example, the grammar of nodes, properties and labels easily maps to
pure YAML, but phandles and overlay trees have semantic meaning that
must be resolved by DT specific code. I’ll respond to you’re specific
examples below...

>
> For instance there’s no way a normal YAML parser won’t choke with something 
> like
>
> / {
>   foo;
> };
>
> / {
>   bar;
> };
>
> Which is a common idiom in DTS files.

That’s only true when the top level of nodes is encoded as a map, but
it shouldn’t be. It should be a list instead, for two reasons. First,
order matters for the top level. Subsequent top level trees have to be
applied in order to fully resolve the tree, but a map doesn’t preserve
the ordering. Second, as you rightly point out, the same name can be
used for multiple top level trees. So, the encoding should be a list
with each list entry containing the node path {/path,}, and the
node data. One possible way to do this is with a tuple:

treedata:
- - /
  - foo:
- - /
  - bar:

Another option is to add a special property in the node to contain the name:

treedata:
- $path: /
  foo:
- $path: /
  bar:

Personally, I prefer the special property approach. That would also be
a good way to encode labels

> The decision to use the YAML anchors and references was more borne out of a 
> desire
> to sort of match conceptually the labels and references of DTS. It’s not a big
> issue to switch to something different.

I think it would be a good idea. One reason for defining a YAML
encoding is to be able to use existing tools to work with the data.
Changing the meaning of anchors and aliases breaks those tools
immediately.

> If we were to force YAML/DT 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
> 
> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>  wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
>> 
>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
>> 
>> Please try it on your platform and report if you encounter any problems.
>> 
>> https://github.com/pantoniou/yamldt
>> 
>> I am eagerly awaiting for your comments.
> 
> Hi Pantelis,
> 
> This is good work. I've played around with it and I'm looking forward
> to chatting next week.
> 

Thanks. I’m looking forward to it too.

> One thing I've done is tried loading the output YAML files into
> another YAML interpreter and the current encoding causes problems.
> Specifically, in yamldt anchors/aliases are being used as a
> replacement for labels/phandles, but that conflicts with the YAML data
> model which defines a reference as a way to make a copy of the data
> appear in another part of the tree. For example, for the following
> snippit:
> 
> intc: intc@1 {
>#interrupt-cells = <1>;
>compatible = "acme,intc";
>reg = <0x1 0x1000>;
>gpio-controller;
> };
> 
> serial@2 {
>compatible = "acme,uart";
>reg = <0x2 0x1000>;
>interrupt-parent = <>;
>interrupts = <5>;
> };
> 
> yamldt will encode this as:
> 
> intc@1: 
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
> 
> serial@2:
>compatible: acme,uart
>reg: [0x2, 0x1000]
>interrupt-parent: *intc
>interrupts: 5
> 
> But, the expected behaviour for a YAML parser is expand the alias
> '*intc' which results in the following structure:
> 
> intc@1: 
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
> 
> serial@2:
>compatible: acme,uart
>reg: [0x2, 0x1000]
>interrupt-parent:
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
>interrupts: 5
> 
> See? It results in the entire interrupt controller node to appear as
> an instance under the interrupt-parent property, when the intention is
> only to create a phandle. yamldt should not redefine the behaviour of
> '*' aliases. Instead, it should use a different indicator, either
> using an explicit !phandle tag, or by replacing '*' with something
> else. I worked around it in my tests by replacing '*' with '$’.

Yes. This is expected. I don’t think pure YAML form is a good match for all
the crazy things that are possible (and actually used in practice) with DTS.

For instance there’s no way a normal YAML parser won’t choke with something like

/ {
  foo;
};

/ {
  bar;
};

Which is a common idiom in DTS files.

The decision to use the YAML anchors and references was more borne out of a 
desire
to sort of match conceptually the labels and references of DTS. It’s not a big
issue to switch to something different.  

If we were to force YAML/DT file to be regular YAML files parseable by 
everything
we’d have to rethink a lot of conventions that DT files currently use and I’m 
afraid
a machine translation from DTS to YAML might not be feasible then.

OTOH moving to that model might make it possible to use YAML constructs that 
are not
mapped at all to DTS. For instance

- foo: true
  bar: “string”

- foo: false
  bar: “another-string”

is not possible to be mapped to a DT node/property structure right now.

> 

> Plus, it would be useful to use normal YAML anchors/aliases for
> creating node templates. For example:
> 
> serial-template:  . # The anchor for the template
>compatible: acme,uart
>interrupt-parent: *intc
> 
> root:
>serial@2:
><<: *acme-uart   # Alias node merged into serial@2
>interrupts: 5
>reg: [0x2, 0x1000]
>serial@3:
><<: *acme-uart   # Alias node merged into serial@3
>interrupts: 5
>reg: [0x3, 0x1000]
> 

Yes, I’m aware of this and I planned to talk to you about it :)
It can be a very powerful way to cut down the churn of DT right now.

It’s not going to be a problem for yamldt to support it (perhaps
adding a new editing tags to allow more fine grained control of the
substitions).
 
> Another problem with anchors/references is YAML seems to require the
> anchor to be defined before the reference, or at least that's what
> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
> should be readily consumable by existing yaml implementations without
> having to do a 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Grant,

> On Oct 20, 2017, at 20:46 , Grant Likely  wrote:
> 
> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
>  wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
>> 
>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
>> 
>> Please try it on your platform and report if you encounter any problems.
>> 
>> https://github.com/pantoniou/yamldt
>> 
>> I am eagerly awaiting for your comments.
> 
> Hi Pantelis,
> 
> This is good work. I've played around with it and I'm looking forward
> to chatting next week.
> 

Thanks. I’m looking forward to it too.

> One thing I've done is tried loading the output YAML files into
> another YAML interpreter and the current encoding causes problems.
> Specifically, in yamldt anchors/aliases are being used as a
> replacement for labels/phandles, but that conflicts with the YAML data
> model which defines a reference as a way to make a copy of the data
> appear in another part of the tree. For example, for the following
> snippit:
> 
> intc: intc@1 {
>#interrupt-cells = <1>;
>compatible = "acme,intc";
>reg = <0x1 0x1000>;
>gpio-controller;
> };
> 
> serial@2 {
>compatible = "acme,uart";
>reg = <0x2 0x1000>;
>interrupt-parent = <>;
>interrupts = <5>;
> };
> 
> yamldt will encode this as:
> 
> intc@1: 
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
> 
> serial@2:
>compatible: acme,uart
>reg: [0x2, 0x1000]
>interrupt-parent: *intc
>interrupts: 5
> 
> But, the expected behaviour for a YAML parser is expand the alias
> '*intc' which results in the following structure:
> 
> intc@1: 
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
> 
> serial@2:
>compatible: acme,uart
>reg: [0x2, 0x1000]
>interrupt-parent:
>"#interrupt-cells": 1
>compatible: acme,intc
>reg: [0x1, 0x1000]
>gpio-controller:
>interrupts: 5
> 
> See? It results in the entire interrupt controller node to appear as
> an instance under the interrupt-parent property, when the intention is
> only to create a phandle. yamldt should not redefine the behaviour of
> '*' aliases. Instead, it should use a different indicator, either
> using an explicit !phandle tag, or by replacing '*' with something
> else. I worked around it in my tests by replacing '*' with '$’.

Yes. This is expected. I don’t think pure YAML form is a good match for all
the crazy things that are possible (and actually used in practice) with DTS.

For instance there’s no way a normal YAML parser won’t choke with something like

/ {
  foo;
};

/ {
  bar;
};

Which is a common idiom in DTS files.

The decision to use the YAML anchors and references was more borne out of a 
desire
to sort of match conceptually the labels and references of DTS. It’s not a big
issue to switch to something different.  

If we were to force YAML/DT file to be regular YAML files parseable by 
everything
we’d have to rethink a lot of conventions that DT files currently use and I’m 
afraid
a machine translation from DTS to YAML might not be feasible then.

OTOH moving to that model might make it possible to use YAML constructs that 
are not
mapped at all to DTS. For instance

- foo: true
  bar: “string”

- foo: false
  bar: “another-string”

is not possible to be mapped to a DT node/property structure right now.

> 

> Plus, it would be useful to use normal YAML anchors/aliases for
> creating node templates. For example:
> 
> serial-template:  . # The anchor for the template
>compatible: acme,uart
>interrupt-parent: *intc
> 
> root:
>serial@2:
><<: *acme-uart   # Alias node merged into serial@2
>interrupts: 5
>reg: [0x2, 0x1000]
>serial@3:
><<: *acme-uart   # Alias node merged into serial@3
>interrupts: 5
>reg: [0x3, 0x1000]
> 

Yes, I’m aware of this and I planned to talk to you about it :)
It can be a very powerful way to cut down the churn of DT right now.

It’s not going to be a problem for yamldt to support it (perhaps
adding a new editing tags to allow more fine grained control of the
substitions).
 
> Another problem with anchors/references is YAML seems to require the
> anchor to be defined before the reference, or at least that's what
> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
> should be readily consumable by existing yaml implementations without
> having to do a lot of customization.
> 

I think this is only true for 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
 wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.
>
> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.
>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.
>
> Please try it on your platform and report if you encounter any problems.
>
> https://github.com/pantoniou/yamldt
>
> I am eagerly awaiting for your comments.

Hi Pantelis,

This is good work. I've played around with it and I'm looking forward
to chatting next week.

One thing I've done is tried loading the output YAML files into
another YAML interpreter and the current encoding causes problems.
Specifically, in yamldt anchors/aliases are being used as a
replacement for labels/phandles, but that conflicts with the YAML data
model which defines a reference as a way to make a copy of the data
appear in another part of the tree. For example, for the following
snippit:

intc: intc@1 {
#interrupt-cells = <1>;
compatible = "acme,intc";
reg = <0x1 0x1000>;
gpio-controller;
};

serial@2 {
compatible = "acme,uart";
reg = <0x2 0x1000>;
interrupt-parent = <>;
interrupts = <5>;
};

yamldt will encode this as:

intc@1: 
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:

serial@2:
compatible: acme,uart
reg: [0x2, 0x1000]
interrupt-parent: *intc
interrupts: 5

But, the expected behaviour for a YAML parser is expand the alias
'*intc' which results in the following structure:

intc@1: 
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:

serial@2:
compatible: acme,uart
reg: [0x2, 0x1000]
interrupt-parent:
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:
interrupts: 5

See? It results in the entire interrupt controller node to appear as
an instance under the interrupt-parent property, when the intention is
only to create a phandle. yamldt should not redefine the behaviour of
'*' aliases. Instead, it should use a different indicator, either
using an explicit !phandle tag, or by replacing '*' with something
else. I worked around it in my tests by replacing '*' with '$'.

Plus, it would be useful to use normal YAML anchors/aliases for
creating node templates. For example:

serial-template:  . # The anchor for the template
compatible: acme,uart
interrupt-parent: *intc

root:
serial@2:
<<: *acme-uart   # Alias node merged into serial@2
interrupts: 5
reg: [0x2, 0x1000]
serial@3:
<<: *acme-uart   # Alias node merged into serial@3
interrupts: 5
reg: [0x3, 0x1000]

Another problem with anchors/references is YAML seems to require the
anchor to be defined before the reference, or at least that's what
pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
should be readily consumable by existing yaml implementations without
having to do a lot of customization.

I'm slightly concerned about using & anchors for labels because it
seems only one anchor can be defined per node, but DTC allows multiple
labels for a single node. This might not be an issue in practice
though. Another implementation issue related to using & anchors is the
YAML spec defines them as an encoding artifact, and parsers can
discard the anchor names after parsing the YAML structure, which is a
problem if we use something like $name to reference an anchor. The
solution might just be that labels need to go into a special property
so they don't disappear from the data stream.

There appears to be no place to put metadata. The root of the tree is
the top level of the YAML structure. There isn't any provision for
having a top level object to hold things like the memreserve map. We
may need a namespace to use for special properties that aren't nodes
or properties.

The encoding differentiates between nodes and properties implicitly
base on whether the contents are a map, or a scalar/list. This does
mean any parser needs to do a bit more work and it may impact what can
be done with validation (I'm not going to talk about validation in
this email though. We'll talk next week.)

Cheers,
g.

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Grant Likely]

On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
 wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.
>
> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.
>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.
>
> Please try it on your platform and report if you encounter any problems.
>
> https://github.com/pantoniou/yamldt
>
> I am eagerly awaiting for your comments.

Hi Pantelis,

This is good work. I've played around with it and I'm looking forward
to chatting next week.

One thing I've done is tried loading the output YAML files into
another YAML interpreter and the current encoding causes problems.
Specifically, in yamldt anchors/aliases are being used as a
replacement for labels/phandles, but that conflicts with the YAML data
model which defines a reference as a way to make a copy of the data
appear in another part of the tree. For example, for the following
snippit:

intc: intc@1 {
#interrupt-cells = <1>;
compatible = "acme,intc";
reg = <0x1 0x1000>;
gpio-controller;
};

serial@2 {
compatible = "acme,uart";
reg = <0x2 0x1000>;
interrupt-parent = <>;
interrupts = <5>;
};

yamldt will encode this as:

intc@1: 
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:

serial@2:
compatible: acme,uart
reg: [0x2, 0x1000]
interrupt-parent: *intc
interrupts: 5

But, the expected behaviour for a YAML parser is expand the alias
'*intc' which results in the following structure:

intc@1: 
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:

serial@2:
compatible: acme,uart
reg: [0x2, 0x1000]
interrupt-parent:
"#interrupt-cells": 1
compatible: acme,intc
reg: [0x1, 0x1000]
gpio-controller:
interrupts: 5

See? It results in the entire interrupt controller node to appear as
an instance under the interrupt-parent property, when the intention is
only to create a phandle. yamldt should not redefine the behaviour of
'*' aliases. Instead, it should use a different indicator, either
using an explicit !phandle tag, or by replacing '*' with something
else. I worked around it in my tests by replacing '*' with '$'.

Plus, it would be useful to use normal YAML anchors/aliases for
creating node templates. For example:

serial-template:  . # The anchor for the template
compatible: acme,uart
interrupt-parent: *intc

root:
serial@2:
<<: *acme-uart   # Alias node merged into serial@2
interrupts: 5
reg: [0x2, 0x1000]
serial@3:
<<: *acme-uart   # Alias node merged into serial@3
interrupts: 5
reg: [0x3, 0x1000]

Another problem with anchors/references is YAML seems to require the
anchor to be defined before the reference, or at least that's what
pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
should be readily consumable by existing yaml implementations without
having to do a lot of customization.

I'm slightly concerned about using & anchors for labels because it
seems only one anchor can be defined per node, but DTC allows multiple
labels for a single node. This might not be an issue in practice
though. Another implementation issue related to using & anchors is the
YAML spec defines them as an encoding artifact, and parsers can
discard the anchor names after parsing the YAML structure, which is a
problem if we use something like $name to reference an anchor. The
solution might just be that labels need to go into a special property
so they don't disappear from the data stream.

There appears to be no place to put metadata. The root of the tree is
the top level of the YAML structure. There isn't any provision for
having a top level object to hold things like the memreserve map. We
may need a namespace to use for special properties that aren't nodes
or properties.

The encoding differentiates between nodes and properties implicitly
base on whether the contents are a map, or a scalar/list. This does
mean any parser needs to do a bit more work and it may impact what can
be done with validation (I'm not going to talk about validation in
this email though. We'll talk next week.)

Cheers,
g.

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Tue, Oct 10, 2017 at 06:19:03PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 10, 2017, at 04:50 , David Gibson  
> > wrote:
> > 
> > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> >> Hi David,
> >> 
> >>> On Oct 9, 2017, at 03:00 , David Gibson  
> >>> wrote:
> >>> 
> >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>  On 10/07/17 03:23, Pantelis Antoniou wrote:
> > Hi Rob,
> > 
> >> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> >> 
> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>  wrote:
> >>> Hi Rob,
>  
>  < snip >
>  
> >>> eBPF is portable, can be serialized after compiling in the schema file
> >>> and can be executed in the kernel.
> >> 
> >> Executing in the kernel is a non-goal for me.
>  
>  Executing in the kernel is an anti-goal for me.
>  
>  We are trying to reduce the device tree footprint inside the kernel,
>  not increase it.
>  
>  99.99% of the validation should be possible statically, in the compile
>  phase.
>  
>  
> >>> By stripping out all documentation related properties and nodes 
> >>> keeping
> >>> only the compiled filters you can generate a dtb blob that passed to
> >>> kernel can be used for verification of all runtime changes in the
> >>> kernel's live tree. eBPF is enforcing an execution model that is 
> >>> 'safe'
> >>> so we can be sure that no foul play is possible.
>  
>  Run time changes can be assumed correct (short of bugs in the overlay
>  application code), if the base tree is validated, the overlay is 
>  validated,
>  and the interface between the live tree and the overlay is a
>  connector.
> >>> 
> >>> In addition, no amount of schema validation can really protect the
> >>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> >>> is "syntactically" correct, which is ambitious, it can't protect
> >>> against a DT which is in the right form, but contains information that
> >>> is simply wrong for the hardware in question.  That can stuff the
> >>> kernel at least as easily as an incorrectly formatted DT.
> >>> 
> >> 
> >> I disagree.
> >> 
> >> There are multiple levels of validation. For now we’re only talking about
> >> binding validation. There can be SoC level validation, board level 
> >> validation,
> >> revision level validation and finally application specific validation.
> >> 
> >> Binding validation is making sure properties/nodes follow the binding 
> >> document.
> >> For instance that for a foo device there’s a mandatory interrupt property.
> >> 
> >> Simplified
> >> 
> >> interrupt = ;
> >> 
> >> Binding validation would ‘catch’ errors like assigning a string or not 
> >> having the
> >> interrupt property available.
> >> 
> >> SoC level validation would list the available interrupt number that a given
> >> SoC would support for that device.
> >> 
> >> For example that interrupt can only take the values 10 or 99 in a given 
> >> SoC.
> >> 
> >> Board level validation would narrow this down even further to a value of 
> >> 10 for
> >> a given board model.
> > 
> >> Similar revision level validation would place further restriction on the 
> >> allowed
> >> configuration.
> >> 
> >> Finally application specific validation could place restriction based on 
> >> the intended
> >> application that piece of hardware is used for. For instance devices that 
> >> should not
> >> exceed a given power budget would have restrictions on the clock frequency 
> >> of the processor
> >> or bus frequencies etc.
> > 
> > This doesn't help.  In order to do this, the validator would need
> > information that's essentially equivalent to the content of DT, at
> > which point there's no point to the DT at all - and you're left with
> > the problem of validating the information that the validator has.
> 
> That would be the case if hardware IP only has a single way to be configured.

Right, and if if there's more than one way, then the validator can't
possibly tell whether the DT has the right one.

DTs must always come from a trusted source, because if they don't,
then you don't need the DT in the first place (you could build your
own).

> The industry standard nowadays is picking reusable IP blocks and integrating 
> them
> together in an SoC. The driver and the binding is common for every platform 
> that uses
> that block, but the allowed configuration varies according to what the 
> hardware
> people uses in a given instance.

> > Fundamentally a validator that's useful *cannot* tell the difference
> > between a correct tree and one which _could_ be correct for some
> > theoretical hardware, but isn't for this particular hardware.
> 
> That’s why there’s reason for a nested hierarchy of bindings IMO.

Nothing about how 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Tue, Oct 10, 2017 at 06:19:03PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 10, 2017, at 04:50 , David Gibson  
> > wrote:
> > 
> > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> >> Hi David,
> >> 
> >>> On Oct 9, 2017, at 03:00 , David Gibson  
> >>> wrote:
> >>> 
> >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>  On 10/07/17 03:23, Pantelis Antoniou wrote:
> > Hi Rob,
> > 
> >> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> >> 
> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>  wrote:
> >>> Hi Rob,
>  
>  < snip >
>  
> >>> eBPF is portable, can be serialized after compiling in the schema file
> >>> and can be executed in the kernel.
> >> 
> >> Executing in the kernel is a non-goal for me.
>  
>  Executing in the kernel is an anti-goal for me.
>  
>  We are trying to reduce the device tree footprint inside the kernel,
>  not increase it.
>  
>  99.99% of the validation should be possible statically, in the compile
>  phase.
>  
>  
> >>> By stripping out all documentation related properties and nodes 
> >>> keeping
> >>> only the compiled filters you can generate a dtb blob that passed to
> >>> kernel can be used for verification of all runtime changes in the
> >>> kernel's live tree. eBPF is enforcing an execution model that is 
> >>> 'safe'
> >>> so we can be sure that no foul play is possible.
>  
>  Run time changes can be assumed correct (short of bugs in the overlay
>  application code), if the base tree is validated, the overlay is 
>  validated,
>  and the interface between the live tree and the overlay is a
>  connector.
> >>> 
> >>> In addition, no amount of schema validation can really protect the
> >>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> >>> is "syntactically" correct, which is ambitious, it can't protect
> >>> against a DT which is in the right form, but contains information that
> >>> is simply wrong for the hardware in question.  That can stuff the
> >>> kernel at least as easily as an incorrectly formatted DT.
> >>> 
> >> 
> >> I disagree.
> >> 
> >> There are multiple levels of validation. For now we’re only talking about
> >> binding validation. There can be SoC level validation, board level 
> >> validation,
> >> revision level validation and finally application specific validation.
> >> 
> >> Binding validation is making sure properties/nodes follow the binding 
> >> document.
> >> For instance that for a foo device there’s a mandatory interrupt property.
> >> 
> >> Simplified
> >> 
> >> interrupt = ;
> >> 
> >> Binding validation would ‘catch’ errors like assigning a string or not 
> >> having the
> >> interrupt property available.
> >> 
> >> SoC level validation would list the available interrupt number that a given
> >> SoC would support for that device.
> >> 
> >> For example that interrupt can only take the values 10 or 99 in a given 
> >> SoC.
> >> 
> >> Board level validation would narrow this down even further to a value of 
> >> 10 for
> >> a given board model.
> > 
> >> Similar revision level validation would place further restriction on the 
> >> allowed
> >> configuration.
> >> 
> >> Finally application specific validation could place restriction based on 
> >> the intended
> >> application that piece of hardware is used for. For instance devices that 
> >> should not
> >> exceed a given power budget would have restrictions on the clock frequency 
> >> of the processor
> >> or bus frequencies etc.
> > 
> > This doesn't help.  In order to do this, the validator would need
> > information that's essentially equivalent to the content of DT, at
> > which point there's no point to the DT at all - and you're left with
> > the problem of validating the information that the validator has.
> 
> That would be the case if hardware IP only has a single way to be configured.

Right, and if if there's more than one way, then the validator can't
possibly tell whether the DT has the right one.

DTs must always come from a trusted source, because if they don't,
then you don't need the DT in the first place (you could build your
own).

> The industry standard nowadays is picking reusable IP blocks and integrating 
> them
> together in an SoC. The driver and the binding is common for every platform 
> that uses
> that block, but the allowed configuration varies according to what the 
> hardware
> people uses in a given instance.

> > Fundamentally a validator that's useful *cannot* tell the difference
> > between a correct tree and one which _could_ be correct for some
> > theoretical hardware, but isn't for this particular hardware.
> 
> That’s why there’s reason for a nested hierarchy of bindings IMO.

Nothing about how you structure the validation can change the basic
fact that there are only two possibilities.  Either:

a) You 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 10, 2017, at 04:50 , David Gibson  wrote:
> 
> On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
>> Hi David,
>> 
>>> On Oct 9, 2017, at 03:00 , David Gibson  wrote:
>>> 
>>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
 On 10/07/17 03:23, Pantelis Antoniou wrote:
> Hi Rob,
> 
>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>> 
>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Rob,
 
 < snip >
 
>>> eBPF is portable, can be serialized after compiling in the schema file
>>> and can be executed in the kernel.
>> 
>> Executing in the kernel is a non-goal for me.
 
 Executing in the kernel is an anti-goal for me.
 
 We are trying to reduce the device tree footprint inside the kernel,
 not increase it.
 
 99.99% of the validation should be possible statically, in the compile
 phase.
 
 
>>> By stripping out all documentation related properties and nodes keeping
>>> only the compiled filters you can generate a dtb blob that passed to
>>> kernel can be used for verification of all runtime changes in the
>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>> so we can be sure that no foul play is possible.
 
 Run time changes can be assumed correct (short of bugs in the overlay
 application code), if the base tree is validated, the overlay is validated,
 and the interface between the live tree and the overlay is a
 connector.
>>> 
>>> In addition, no amount of schema validation can really protect the
>>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
>>> is "syntactically" correct, which is ambitious, it can't protect
>>> against a DT which is in the right form, but contains information that
>>> is simply wrong for the hardware in question.  That can stuff the
>>> kernel at least as easily as an incorrectly formatted DT.
>>> 
>> 
>> I disagree.
>> 
>> There are multiple levels of validation. For now we’re only talking about
>> binding validation. There can be SoC level validation, board level 
>> validation,
>> revision level validation and finally application specific validation.
>> 
>> Binding validation is making sure properties/nodes follow the binding 
>> document.
>> For instance that for a foo device there’s a mandatory interrupt property.
>> 
>> Simplified
>> 
>> interrupt = ;
>> 
>> Binding validation would ‘catch’ errors like assigning a string or not 
>> having the
>> interrupt property available.
>> 
>> SoC level validation would list the available interrupt number that a given
>> SoC would support for that device.
>> 
>> For example that interrupt can only take the values 10 or 99 in a given SoC.
>> 
>> Board level validation would narrow this down even further to a value of 10 
>> for
>> a given board model.
> 
>> Similar revision level validation would place further restriction on the 
>> allowed
>> configuration.
>> 
>> Finally application specific validation could place restriction based on the 
>> intended
>> application that piece of hardware is used for. For instance devices that 
>> should not
>> exceed a given power budget would have restrictions on the clock frequency 
>> of the processor
>> or bus frequencies etc.
> 
> This doesn't help.  In order to do this, the validator would need
> information that's essentially equivalent to the content of DT, at
> which point there's no point to the DT at all - and you're left with
> the problem of validating the information that the validator has.
> 

That would be the case if hardware IP only has a single way to be configured.

The industry standard nowadays is picking reusable IP blocks and integrating 
them
together in an SoC. The driver and the binding is common for every platform 
that uses
that block, but the allowed configuration varies according to what the hardware
people uses in a given instance.

> Fundamentally a validator that's useful *cannot* tell the difference
> between a correct tree and one which _could_ be correct for some
> theoretical hardware, but isn't for this particular hardware.
> 

That’s why there’s reason for a nested hierarchy of bindings IMO.

Completeness of validation schemes can be a differentiating factor when
choosing parts for hardware design. They would sure cut down development time.


> -- 
> David Gibson  | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au| minimalist, thank you.  NOT _the_ 
> _other_
>   | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 10, 2017, at 04:50 , David Gibson  wrote:
> 
> On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
>> Hi David,
>> 
>>> On Oct 9, 2017, at 03:00 , David Gibson  wrote:
>>> 
>>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
 On 10/07/17 03:23, Pantelis Antoniou wrote:
> Hi Rob,
> 
>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>> 
>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Rob,
 
 < snip >
 
>>> eBPF is portable, can be serialized after compiling in the schema file
>>> and can be executed in the kernel.
>> 
>> Executing in the kernel is a non-goal for me.
 
 Executing in the kernel is an anti-goal for me.
 
 We are trying to reduce the device tree footprint inside the kernel,
 not increase it.
 
 99.99% of the validation should be possible statically, in the compile
 phase.
 
 
>>> By stripping out all documentation related properties and nodes keeping
>>> only the compiled filters you can generate a dtb blob that passed to
>>> kernel can be used for verification of all runtime changes in the
>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>> so we can be sure that no foul play is possible.
 
 Run time changes can be assumed correct (short of bugs in the overlay
 application code), if the base tree is validated, the overlay is validated,
 and the interface between the live tree and the overlay is a
 connector.
>>> 
>>> In addition, no amount of schema validation can really protect the
>>> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
>>> is "syntactically" correct, which is ambitious, it can't protect
>>> against a DT which is in the right form, but contains information that
>>> is simply wrong for the hardware in question.  That can stuff the
>>> kernel at least as easily as an incorrectly formatted DT.
>>> 
>> 
>> I disagree.
>> 
>> There are multiple levels of validation. For now we’re only talking about
>> binding validation. There can be SoC level validation, board level 
>> validation,
>> revision level validation and finally application specific validation.
>> 
>> Binding validation is making sure properties/nodes follow the binding 
>> document.
>> For instance that for a foo device there’s a mandatory interrupt property.
>> 
>> Simplified
>> 
>> interrupt = ;
>> 
>> Binding validation would ‘catch’ errors like assigning a string or not 
>> having the
>> interrupt property available.
>> 
>> SoC level validation would list the available interrupt number that a given
>> SoC would support for that device.
>> 
>> For example that interrupt can only take the values 10 or 99 in a given SoC.
>> 
>> Board level validation would narrow this down even further to a value of 10 
>> for
>> a given board model.
> 
>> Similar revision level validation would place further restriction on the 
>> allowed
>> configuration.
>> 
>> Finally application specific validation could place restriction based on the 
>> intended
>> application that piece of hardware is used for. For instance devices that 
>> should not
>> exceed a given power budget would have restrictions on the clock frequency 
>> of the processor
>> or bus frequencies etc.
> 
> This doesn't help.  In order to do this, the validator would need
> information that's essentially equivalent to the content of DT, at
> which point there's no point to the DT at all - and you're left with
> the problem of validating the information that the validator has.
> 

That would be the case if hardware IP only has a single way to be configured.

The industry standard nowadays is picking reusable IP blocks and integrating 
them
together in an SoC. The driver and the binding is common for every platform 
that uses
that block, but the allowed configuration varies according to what the hardware
people uses in a given instance.

> Fundamentally a validator that's useful *cannot* tell the difference
> between a correct tree and one which _could_ be correct for some
> theoretical hardware, but isn't for this particular hardware.
> 

That’s why there’s reason for a nested hierarchy of bindings IMO.

Completeness of validation schemes can be a differentiating factor when
choosing parts for hardware design. They would sure cut down development time.


> -- 
> David Gibson  | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au| minimalist, thank you.  NOT _the_ 
> _other_
>   | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 9, 2017, at 03:00 , David Gibson  wrote:
> > 
> > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> >> On 10/07/17 03:23, Pantelis Antoniou wrote:
> >>> Hi Rob,
> >>> 
>  On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>  
>  On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>   wrote:
> > Hi Rob,
> >> 
> >> < snip >
> >> 
> > eBPF is portable, can be serialized after compiling in the schema file
> > and can be executed in the kernel.
>  
>  Executing in the kernel is a non-goal for me.
> >> 
> >> Executing in the kernel is an anti-goal for me.
> >> 
> >> We are trying to reduce the device tree footprint inside the kernel,
> >> not increase it.
> >> 
> >> 99.99% of the validation should be possible statically, in the compile
> >> phase.
> >> 
> >> 
> > By stripping out all documentation related properties and nodes keeping
> > only the compiled filters you can generate a dtb blob that passed to
> > kernel can be used for verification of all runtime changes in the
> > kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> > so we can be sure that no foul play is possible.
> >> 
> >> Run time changes can be assumed correct (short of bugs in the overlay
> >> application code), if the base tree is validated, the overlay is validated,
> >> and the interface between the live tree and the overlay is a
> >> connector.
> > 
> > In addition, no amount of schema validation can really protect the
> > kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> > is "syntactically" correct, which is ambitious, it can't protect
> > against a DT which is in the right form, but contains information that
> > is simply wrong for the hardware in question.  That can stuff the
> > kernel at least as easily as an incorrectly formatted DT.
> > 
> 
> I disagree.
> 
> There are multiple levels of validation. For now we’re only talking about
> binding validation. There can be SoC level validation, board level validation,
> revision level validation and finally application specific validation.
> 
> Binding validation is making sure properties/nodes follow the binding 
> document.
> For instance that for a foo device there’s a mandatory interrupt property.
> 
> Simplified
> 
> interrupt = ;
> 
> Binding validation would ‘catch’ errors like assigning a string or not having 
> the
> interrupt property available.
> 
> SoC level validation would list the available interrupt number that a given
> SoC would support for that device.
> 
> For example that interrupt can only take the values 10 or 99 in a given SoC.
> 
> Board level validation would narrow this down even further to a value of 10 
> for
> a given board model.

> Similar revision level validation would place further restriction on the 
> allowed
> configuration.
> 
> Finally application specific validation could place restriction based on the 
> intended
> application that piece of hardware is used for. For instance devices that 
> should not
> exceed a given power budget would have restrictions on the clock frequency of 
> the processor
> or bus frequencies etc.

This doesn't help.  In order to do this, the validator would need
information that's essentially equivalent to the content of DT, at
which point there's no point to the DT at all - and you're left with
the problem of validating the information that the validator has.

Fundamentally a validator that's useful *cannot* tell the difference
between a correct tree and one which _could_ be correct for some
theoretical hardware, but isn't for this particular hardware.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote:
> Hi David,
> 
> > On Oct 9, 2017, at 03:00 , David Gibson  wrote:
> > 
> > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> >> On 10/07/17 03:23, Pantelis Antoniou wrote:
> >>> Hi Rob,
> >>> 
>  On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>  
>  On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>   wrote:
> > Hi Rob,
> >> 
> >> < snip >
> >> 
> > eBPF is portable, can be serialized after compiling in the schema file
> > and can be executed in the kernel.
>  
>  Executing in the kernel is a non-goal for me.
> >> 
> >> Executing in the kernel is an anti-goal for me.
> >> 
> >> We are trying to reduce the device tree footprint inside the kernel,
> >> not increase it.
> >> 
> >> 99.99% of the validation should be possible statically, in the compile
> >> phase.
> >> 
> >> 
> > By stripping out all documentation related properties and nodes keeping
> > only the compiled filters you can generate a dtb blob that passed to
> > kernel can be used for verification of all runtime changes in the
> > kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> > so we can be sure that no foul play is possible.
> >> 
> >> Run time changes can be assumed correct (short of bugs in the overlay
> >> application code), if the base tree is validated, the overlay is validated,
> >> and the interface between the live tree and the overlay is a
> >> connector.
> > 
> > In addition, no amount of schema validation can really protect the
> > kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> > is "syntactically" correct, which is ambitious, it can't protect
> > against a DT which is in the right form, but contains information that
> > is simply wrong for the hardware in question.  That can stuff the
> > kernel at least as easily as an incorrectly formatted DT.
> > 
> 
> I disagree.
> 
> There are multiple levels of validation. For now we’re only talking about
> binding validation. There can be SoC level validation, board level validation,
> revision level validation and finally application specific validation.
> 
> Binding validation is making sure properties/nodes follow the binding 
> document.
> For instance that for a foo device there’s a mandatory interrupt property.
> 
> Simplified
> 
> interrupt = ;
> 
> Binding validation would ‘catch’ errors like assigning a string or not having 
> the
> interrupt property available.
> 
> SoC level validation would list the available interrupt number that a given
> SoC would support for that device.
> 
> For example that interrupt can only take the values 10 or 99 in a given SoC.
> 
> Board level validation would narrow this down even further to a value of 10 
> for
> a given board model.

> Similar revision level validation would place further restriction on the 
> allowed
> configuration.
> 
> Finally application specific validation could place restriction based on the 
> intended
> application that piece of hardware is used for. For instance devices that 
> should not
> exceed a given power budget would have restrictions on the clock frequency of 
> the processor
> or bus frequencies etc.

This doesn't help.  In order to do this, the validator would need
information that's essentially equivalent to the content of DT, at
which point there's no point to the DT at all - and you're left with
the problem of validating the information that the validator has.

Fundamentally a validator that's useful *cannot* tell the difference
between a correct tree and one which _could_ be correct for some
theoretical hardware, but isn't for this particular hardware.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 9, 2017, at 03:00 , David Gibson  wrote:
> 
> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>> On 10/07/17 03:23, Pantelis Antoniou wrote:
>>> Hi Rob,
>>> 
 On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
 
 On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
  wrote:
> Hi Rob,
>> 
>> < snip >
>> 
> eBPF is portable, can be serialized after compiling in the schema file
> and can be executed in the kernel.
 
 Executing in the kernel is a non-goal for me.
>> 
>> Executing in the kernel is an anti-goal for me.
>> 
>> We are trying to reduce the device tree footprint inside the kernel,
>> not increase it.
>> 
>> 99.99% of the validation should be possible statically, in the compile
>> phase.
>> 
>> 
> By stripping out all documentation related properties and nodes keeping
> only the compiled filters you can generate a dtb blob that passed to
> kernel can be used for verification of all runtime changes in the
> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> so we can be sure that no foul play is possible.
>> 
>> Run time changes can be assumed correct (short of bugs in the overlay
>> application code), if the base tree is validated, the overlay is validated,
>> and the interface between the live tree and the overlay is a
>> connector.
> 
> In addition, no amount of schema validation can really protect the
> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> is "syntactically" correct, which is ambitious, it can't protect
> against a DT which is in the right form, but contains information that
> is simply wrong for the hardware in question.  That can stuff the
> kernel at least as easily as an incorrectly formatted DT.
> 

I disagree.

There are multiple levels of validation. For now we’re only talking about
binding validation. There can be SoC level validation, board level validation,
revision level validation and finally application specific validation.

Binding validation is making sure properties/nodes follow the binding document.
For instance that for a foo device there’s a mandatory interrupt property.

Simplified

interrupt = ;

Binding validation would ‘catch’ errors like assigning a string or not having 
the
interrupt property available.

SoC level validation would list the available interrupt number that a given
SoC would support for that device.

For example that interrupt can only take the values 10 or 99 in a given SoC.

Board level validation would narrow this down even further to a value of 10 for
a given board model.

Similar revision level validation would place further restriction on the allowed
configuration.

Finally application specific validation could place restriction based on the 
intended
application that piece of hardware is used for. For instance devices that 
should not
exceed a given power budget would have restrictions on the clock frequency of 
the processor
or bus frequencies etc.

> -- 
> David Gibson  | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au| minimalist, thank you.  NOT _the_ 
> _other_
>   | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi David,

> On Oct 9, 2017, at 03:00 , David Gibson  wrote:
> 
> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
>> On 10/07/17 03:23, Pantelis Antoniou wrote:
>>> Hi Rob,
>>> 
 On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
 
 On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
  wrote:
> Hi Rob,
>> 
>> < snip >
>> 
> eBPF is portable, can be serialized after compiling in the schema file
> and can be executed in the kernel.
 
 Executing in the kernel is a non-goal for me.
>> 
>> Executing in the kernel is an anti-goal for me.
>> 
>> We are trying to reduce the device tree footprint inside the kernel,
>> not increase it.
>> 
>> 99.99% of the validation should be possible statically, in the compile
>> phase.
>> 
>> 
> By stripping out all documentation related properties and nodes keeping
> only the compiled filters you can generate a dtb blob that passed to
> kernel can be used for verification of all runtime changes in the
> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> so we can be sure that no foul play is possible.
>> 
>> Run time changes can be assumed correct (short of bugs in the overlay
>> application code), if the base tree is validated, the overlay is validated,
>> and the interface between the live tree and the overlay is a
>> connector.
> 
> In addition, no amount of schema validation can really protect the
> kernel from a bad DT.  Even if the schemas can 100% verify that the DT
> is "syntactically" correct, which is ambitious, it can't protect
> against a DT which is in the right form, but contains information that
> is simply wrong for the hardware in question.  That can stuff the
> kernel at least as easily as an incorrectly formatted DT.
> 

I disagree.

There are multiple levels of validation. For now we’re only talking about
binding validation. There can be SoC level validation, board level validation,
revision level validation and finally application specific validation.

Binding validation is making sure properties/nodes follow the binding document.
For instance that for a foo device there’s a mandatory interrupt property.

Simplified

interrupt = ;

Binding validation would ‘catch’ errors like assigning a string or not having 
the
interrupt property available.

SoC level validation would list the available interrupt number that a given
SoC would support for that device.

For example that interrupt can only take the values 10 or 99 in a given SoC.

Board level validation would narrow this down even further to a value of 10 for
a given board model.

Similar revision level validation would place further restriction on the allowed
configuration.

Finally application specific validation could place restriction based on the 
intended
application that piece of hardware is used for. For instance devices that 
should not
exceed a given power budget would have restrictions on the clock frequency of 
the processor
or bus frequencies etc.

> -- 
> David Gibson  | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au| minimalist, thank you.  NOT _the_ 
> _other_
>   | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Frank,

> On Oct 9, 2017, at 02:08 , Frank Rowand  wrote:
> 
> On 10/07/17 03:23, Pantelis Antoniou wrote:
>> Hi Rob,
>> 
>>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>>> 
>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>>  wrote:
 Hi Rob,
> 
> < snip >
> 
 eBPF is portable, can be serialized after compiling in the schema file
 and can be executed in the kernel.
>>> 
>>> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 

That’s not possible when you dynamically alter the tree at runtime.

> 
 By stripping out all documentation related properties and nodes keeping
 only the compiled filters you can generate a dtb blob that passed to
 kernel can be used for verification of all runtime changes in the
 kernel's live tree. eBPF is enforcing an execution model that is 'safe'
 so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a connector.
> 

You can validate the base tree statically but not the overlays.

In fact a large percentage of overlay usage simply modifies a status property
to turn on or off a device. There is nothing to validate in such case.

The portable connector is still a long ways off and I haven’t seen anything that
could handle something trickier that a few GPIOs and I2C or SPI busses.

My goal is something that works covering all the cases without any surprising
gotchas.

> 
>>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>>> sign them like you would for the kernel or modules.
>>> 
>> 
>> That’s a problem when deploying; the runtime validation is for making sure
>> developers don’t get bogged down chasing problems when working on their
>> own platforms/drivers.
>> 
>> We have absolutely zero checks for stopping badly configured DT blobs
>> hanging the kernel. With runtime validation a bug that might take a few
>> days to figure out can be cut down to a few minutes.
> 
> Same reply as above.
> 
> 
 That means that you can a) run it at boot-time, verifying the dtb blob
 passed by the bootloader for errors (potentially disabling devices
 that their nodes fail) and b) run it when applying overlays to reject
 any that result in an invalid tree.
>>> 
>>> Let's get verification at build time working first, then we can worry
>>> about something like this.
> 
> < snip >
> 

Right, let’s get build verification working first.

> -Frank

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Frank,

> On Oct 9, 2017, at 02:08 , Frank Rowand  wrote:
> 
> On 10/07/17 03:23, Pantelis Antoniou wrote:
>> Hi Rob,
>> 
>>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>>> 
>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>>  wrote:
 Hi Rob,
> 
> < snip >
> 
 eBPF is portable, can be serialized after compiling in the schema file
 and can be executed in the kernel.
>>> 
>>> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 

That’s not possible when you dynamically alter the tree at runtime.

> 
 By stripping out all documentation related properties and nodes keeping
 only the compiled filters you can generate a dtb blob that passed to
 kernel can be used for verification of all runtime changes in the
 kernel's live tree. eBPF is enforcing an execution model that is 'safe'
 so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a connector.
> 

You can validate the base tree statically but not the overlays.

In fact a large percentage of overlay usage simply modifies a status property
to turn on or off a device. There is nothing to validate in such case.

The portable connector is still a long ways off and I haven’t seen anything that
could handle something trickier that a few GPIOs and I2C or SPI busses.

My goal is something that works covering all the cases without any surprising
gotchas.

> 
>>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>>> sign them like you would for the kernel or modules.
>>> 
>> 
>> That’s a problem when deploying; the runtime validation is for making sure
>> developers don’t get bogged down chasing problems when working on their
>> own platforms/drivers.
>> 
>> We have absolutely zero checks for stopping badly configured DT blobs
>> hanging the kernel. With runtime validation a bug that might take a few
>> days to figure out can be cut down to a few minutes.
> 
> Same reply as above.
> 
> 
 That means that you can a) run it at boot-time, verifying the dtb blob
 passed by the bootloader for errors (potentially disabling devices
 that their nodes fail) and b) run it when applying overlays to reject
 any that result in an invalid tree.
>>> 
>>> Let's get verification at build time working first, then we can worry
>>> about something like this.
> 
> < snip >
> 

Right, let’s get build verification working first.

> -Frank

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> On 10/07/17 03:23, Pantelis Antoniou wrote:
> > Hi Rob,
> > 
> >> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> >>
> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>  wrote:
> >>> Hi Rob,
> 
> < snip >
> 
> >>> eBPF is portable, can be serialized after compiling in the schema file
> >>> and can be executed in the kernel.
> >>
> >> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 
> 
> >>> By stripping out all documentation related properties and nodes keeping
> >>> only the compiled filters you can generate a dtb blob that passed to
> >>> kernel can be used for verification of all runtime changes in the
> >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> >>> so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a
> connector.

In addition, no amount of schema validation can really protect the
kernel from a bad DT.  Even if the schemas can 100% verify that the DT
is "syntactically" correct, which is ambitious, it can't protect
against a DT which is in the right form, but contains information that
is simply wrong for the hardware in question.  That can stuff the
kernel at least as easily as an incorrectly formatted DT.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [RFC] yamldt v0.5, now a DTS compiler too

[David Gibson]

On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote:
> On 10/07/17 03:23, Pantelis Antoniou wrote:
> > Hi Rob,
> > 
> >> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> >>
> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
> >>  wrote:
> >>> Hi Rob,
> 
> < snip >
> 
> >>> eBPF is portable, can be serialized after compiling in the schema file
> >>> and can be executed in the kernel.
> >>
> >> Executing in the kernel is a non-goal for me.
> 
> Executing in the kernel is an anti-goal for me.
> 
> We are trying to reduce the device tree footprint inside the kernel,
> not increase it.
> 
> 99.99% of the validation should be possible statically, in the compile
> phase.
> 
> 
> >>> By stripping out all documentation related properties and nodes keeping
> >>> only the compiled filters you can generate a dtb blob that passed to
> >>> kernel can be used for verification of all runtime changes in the
> >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
> >>> so we can be sure that no foul play is possible.
> 
> Run time changes can be assumed correct (short of bugs in the overlay
> application code), if the base tree is validated, the overlay is validated,
> and the interface between the live tree and the overlay is a
> connector.

In addition, no amount of schema validation can really protect the
kernel from a bad DT.  Even if the schemas can 100% verify that the DT
is "syntactically" correct, which is ambitious, it can't protect
against a DT which is in the right form, but contains information that
is simply wrong for the hardware in question.  That can stuff the
kernel at least as easily as an incorrectly formatted DT.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Frank Rowand]

On 10/07/17 03:23, Pantelis Antoniou wrote:
> Hi Rob,
> 
>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>>
>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Rob,

< snip >

>>> eBPF is portable, can be serialized after compiling in the schema file
>>> and can be executed in the kernel.
>>
>> Executing in the kernel is a non-goal for me.

Executing in the kernel is an anti-goal for me.

We are trying to reduce the device tree footprint inside the kernel,
not increase it.

99.99% of the validation should be possible statically, in the compile
phase.


>>> By stripping out all documentation related properties and nodes keeping
>>> only the compiled filters you can generate a dtb blob that passed to
>>> kernel can be used for verification of all runtime changes in the
>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>> so we can be sure that no foul play is possible.

Run time changes can be assumed correct (short of bugs in the overlay
application code), if the base tree is validated, the overlay is validated,
and the interface between the live tree and the overlay is a connector.
 

>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>> sign them like you would for the kernel or modules.
>>
> 
> That’s a problem when deploying; the runtime validation is for making sure
> developers don’t get bogged down chasing problems when working on their
> own platforms/drivers.
> 
> We have absolutely zero checks for stopping badly configured DT blobs
> hanging the kernel. With runtime validation a bug that might take a few
> days to figure out can be cut down to a few minutes.

Same reply as above.


>>> That means that you can a) run it at boot-time, verifying the dtb blob
>>> passed by the bootloader for errors (potentially disabling devices
>>> that their nodes fail) and b) run it when applying overlays to reject
>>> any that result in an invalid tree.
>>
>> Let's get verification at build time working first, then we can worry
>> about something like this.

< snip >

-Frank

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Frank Rowand]

On 10/07/17 03:23, Pantelis Antoniou wrote:
> Hi Rob,
> 
>> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
>>
>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>>  wrote:
>>> Hi Rob,

< snip >

>>> eBPF is portable, can be serialized after compiling in the schema file
>>> and can be executed in the kernel.
>>
>> Executing in the kernel is a non-goal for me.

Executing in the kernel is an anti-goal for me.

We are trying to reduce the device tree footprint inside the kernel,
not increase it.

99.99% of the validation should be possible statically, in the compile
phase.


>>> By stripping out all documentation related properties and nodes keeping
>>> only the compiled filters you can generate a dtb blob that passed to
>>> kernel can be used for verification of all runtime changes in the
>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe'
>>> so we can be sure that no foul play is possible.

Run time changes can be assumed correct (short of bugs in the overlay
application code), if the base tree is validated, the overlay is validated,
and the interface between the live tree and the overlay is a connector.
 

>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just
>> sign them like you would for the kernel or modules.
>>
> 
> That’s a problem when deploying; the runtime validation is for making sure
> developers don’t get bogged down chasing problems when working on their
> own platforms/drivers.
> 
> We have absolutely zero checks for stopping badly configured DT blobs
> hanging the kernel. With runtime validation a bug that might take a few
> days to figure out can be cut down to a few minutes.

Same reply as above.


>>> That means that you can a) run it at boot-time, verifying the dtb blob
>>> passed by the bootloader for errors (potentially disabling devices
>>> that their nodes fail) and b) run it when applying overlays to reject
>>> any that result in an invalid tree.
>>
>> Let's get verification at build time working first, then we can worry
>> about something like this.

< snip >

-Frank

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> 
> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>  wrote:
>> Hi Rob,
>> 
>> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>>>  wrote:
 Hi Rob,
 
 On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>  wrote:
>> Hi Rob,
>> 
>> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>>>  wrote:
 Hello again,
 
 Significant progress has been made on yamldt and is now capable of
 not only generating yaml from DTS source but also compiling DTS sources
 and being almost fully compatible with DTC.
>>> 
>>> Can you quantify "almost"?
>>> 
 Compiling the kernel's DTBs using yamldt is as simple as using a
 DTC=yamldt.
>>> 
>>> Good.
>>> 
 
 Error reporting is accurate and validation against a YAML based schema
 works as well. In a short while I will begin posting patches with
 fixes on bindings and DTS files in the kernel.
>>> 
>>> What I would like to see is the schema format posted for review.
>>> 
>> 
>> I'm including the skeleton.yaml binding which is the template for
>> the bindings and a board-example.yaml binding for a top level binding.
>> 
>>> I would also like to see the bindings for top-level compatible strings
>>> (aka boards) as an example. That's something that's simple enough that
>>> I'd think we could agree on a format and start moving towards defining
>>> board bindings that way.
>>> 
>> 
>> Note there is some line wrapping I'm including a link
>> to the github repo of the files:
>> 
>> 
>> The skeleton.yaml
>> 
>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> 
>> %YAML 1.1
>> ---
>> # The name of the binding is first
>> # The anchor is put there for use by others
>> skeleton: 
> 
> This and "id" seem redundant.
> 
 
 Indeed.
>>> 
>>> One other thing, "skeleton" is a bit weird as a key. It can't be
>>> validated. All the other keys are standard words. I could write
>>> "skeloton" by mistake and I guess I'd only find the mistake when
>>> something inherits it. That's somewhat true with id, but we can at
>>> least check "id" is present and that it's value is unique among all
>>> other id values.
>>> 
>> 
>> We can keep id and check that it matches the name of the enclosing node.
>> That way you can be sure that there's no error.
>> 
>> But it's a bit weird cause this is similar to declaring a function name
>> with a typo. You won't find out until you use it.
>> 
 
>>  version: 1
>> 
>>  id: skel-device
>> 
>>  title: >
>>Skeleton Device
>> 
>>  maintainer:
>>name: Skeleton Person 
>> 
>>  description: >
>>The Skeleton Device binding represents the SK11 device produced by
>>the Skeleton Corporation. The binding can also support compatible
>>clones made by second source vendors.
>> 
>>  # The class is an optional property that declares this
>>  # binding as part of a larger set
>>  # Multiple definitions are possible
>>  class: [ device, spi-device ]
>> 
>>  # This binding inherits property characteristics from the generic
>>  # spi-slave binding
>>  # Note that the notation is standard yaml reference
>>  inherits: *spi-slave
>> 
>>  # virtual bindings do not generate checkers
>>  virtual: true
> 
> virtual is an overloaded term.
> 
 
 OK, what term should I use that this binding should not be instantiated
 as a checker, only be used by other bindings when inherited?
>>> 
>>> checks: true?
>>> 
>>> I'd really like to avoid having to decide and drop this, but I don't
>>> really get why it is needed.
>>> 
>> 
>> It is needed because otherwise checker filters will be generated for
>> the template bindings that while they won't be executed they will be
>> compiled and take up space in the schema.
> 
> Size is not a problem I have. That can come later if needed.
> 
>>  # each property is defined by each name
>>  properties:
>> 
>># The compatible property is a reserved name. The type is always
>> "string"
>># and should not be repeated device binding.
>>compatible:
>>  category: required# required property
>>  type: strseq  # is a sequence of strings
>> 
>>  description: >
>>FX11 is a clone of the original SK11 device
>> 
>>  # v 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

> On Oct 6, 2017, at 16:55 , Rob Herring  wrote:
> 
> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
>  wrote:
>> Hi Rob,
>> 
>> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>>>  wrote:
 Hi Rob,
 
 On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>  wrote:
>> Hi Rob,
>> 
>> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>>>  wrote:
 Hello again,
 
 Significant progress has been made on yamldt and is now capable of
 not only generating yaml from DTS source but also compiling DTS sources
 and being almost fully compatible with DTC.
>>> 
>>> Can you quantify "almost"?
>>> 
 Compiling the kernel's DTBs using yamldt is as simple as using a
 DTC=yamldt.
>>> 
>>> Good.
>>> 
 
 Error reporting is accurate and validation against a YAML based schema
 works as well. In a short while I will begin posting patches with
 fixes on bindings and DTS files in the kernel.
>>> 
>>> What I would like to see is the schema format posted for review.
>>> 
>> 
>> I'm including the skeleton.yaml binding which is the template for
>> the bindings and a board-example.yaml binding for a top level binding.
>> 
>>> I would also like to see the bindings for top-level compatible strings
>>> (aka boards) as an example. That's something that's simple enough that
>>> I'd think we could agree on a format and start moving towards defining
>>> board bindings that way.
>>> 
>> 
>> Note there is some line wrapping I'm including a link
>> to the github repo of the files:
>> 
>> 
>> The skeleton.yaml
>> 
>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> 
>> %YAML 1.1
>> ---
>> # The name of the binding is first
>> # The anchor is put there for use by others
>> skeleton: 
> 
> This and "id" seem redundant.
> 
 
 Indeed.
>>> 
>>> One other thing, "skeleton" is a bit weird as a key. It can't be
>>> validated. All the other keys are standard words. I could write
>>> "skeloton" by mistake and I guess I'd only find the mistake when
>>> something inherits it. That's somewhat true with id, but we can at
>>> least check "id" is present and that it's value is unique among all
>>> other id values.
>>> 
>> 
>> We can keep id and check that it matches the name of the enclosing node.
>> That way you can be sure that there's no error.
>> 
>> But it's a bit weird cause this is similar to declaring a function name
>> with a typo. You won't find out until you use it.
>> 
 
>>  version: 1
>> 
>>  id: skel-device
>> 
>>  title: >
>>Skeleton Device
>> 
>>  maintainer:
>>name: Skeleton Person 
>> 
>>  description: >
>>The Skeleton Device binding represents the SK11 device produced by
>>the Skeleton Corporation. The binding can also support compatible
>>clones made by second source vendors.
>> 
>>  # The class is an optional property that declares this
>>  # binding as part of a larger set
>>  # Multiple definitions are possible
>>  class: [ device, spi-device ]
>> 
>>  # This binding inherits property characteristics from the generic
>>  # spi-slave binding
>>  # Note that the notation is standard yaml reference
>>  inherits: *spi-slave
>> 
>>  # virtual bindings do not generate checkers
>>  virtual: true
> 
> virtual is an overloaded term.
> 
 
 OK, what term should I use that this binding should not be instantiated
 as a checker, only be used by other bindings when inherited?
>>> 
>>> checks: true?
>>> 
>>> I'd really like to avoid having to decide and drop this, but I don't
>>> really get why it is needed.
>>> 
>> 
>> It is needed because otherwise checker filters will be generated for
>> the template bindings that while they won't be executed they will be
>> compiled and take up space in the schema.
> 
> Size is not a problem I have. That can come later if needed.
> 
>>  # each property is defined by each name
>>  properties:
>> 
>># The compatible property is a reserved name. The type is always
>> "string"
>># and should not be repeated device binding.
>>compatible:
>>  category: required# required property
>>  type: strseq  # is a sequence of strings
>> 
>>  description: >
>>FX11 is a clone of the original SK11 device
>> 
>>  # v is always the name of the value of the property
>>  # np is passed to the checker and is the current
>>  # node pointer. We can access properties and 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>>  wrote:
>> > Hi Rob,
>> >
>> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>> >>  wrote:
>> >> > Hi Rob,
>> >> >
>> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >> >>  wrote:
>> >> >> > Hello again,
>> >> >> >
>> >> >> > Significant progress has been made on yamldt and is now capable of
>> >> >> > not only generating yaml from DTS source but also compiling DTS 
>> >> >> > sources
>> >> >> > and being almost fully compatible with DTC.
>> >> >>
>> >> >> Can you quantify "almost"?
>> >> >>
>> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> >> > DTC=yamldt.
>> >> >>
>> >> >> Good.
>> >> >>
>> >> >> >
>> >> >> > Error reporting is accurate and validation against a YAML based 
>> >> >> > schema
>> >> >> > works as well. In a short while I will begin posting patches with
>> >> >> > fixes on bindings and DTS files in the kernel.
>> >> >>
>> >> >> What I would like to see is the schema format posted for review.
>> >> >>
>> >> >
>> >> > I'm including the skeleton.yaml binding which is the template for
>> >> > the bindings and a board-example.yaml binding for a top level binding.
>> >> >
>> >> >> I would also like to see the bindings for top-level compatible strings
>> >> >> (aka boards) as an example. That's something that's simple enough that
>> >> >> I'd think we could agree on a format and start moving towards defining
>> >> >> board bindings that way.
>> >> >>
>> >> >
>> >> > Note there is some line wrapping I'm including a link
>> >> > to the github repo of the files:
>> >> >
>> >> >
>> >> > The skeleton.yaml
>> >> >
>> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >> >
>> >> > %YAML 1.1
>> >> > ---
>> >> > # The name of the binding is first
>> >> > # The anchor is put there for use by others
>> >> > skeleton: 
>> >>
>> >> This and "id" seem redundant.
>> >>
>> >
>> > Indeed.
>>
>> One other thing, "skeleton" is a bit weird as a key. It can't be
>> validated. All the other keys are standard words. I could write
>> "skeloton" by mistake and I guess I'd only find the mistake when
>> something inherits it. That's somewhat true with id, but we can at
>> least check "id" is present and that it's value is unique among all
>> other id values.
>>
>
> We can keep id and check that it matches the name of the enclosing node.
> That way you can be sure that there's no error.
>
> But it's a bit weird cause this is similar to declaring a function name
> with a typo. You won't find out until you use it.
>
>> >
>> >> >   version: 1
>> >> >
>> >> >   id: skel-device
>> >> >
>> >> >   title: >
>> >> > Skeleton Device
>> >> >
>> >> >   maintainer:
>> >> > name: Skeleton Person 
>> >> >
>> >> >   description: >
>> >> > The Skeleton Device binding represents the SK11 device produced by
>> >> > the Skeleton Corporation. The binding can also support compatible
>> >> > clones made by second source vendors.
>> >> >
>> >> >   # The class is an optional property that declares this
>> >> >   # binding as part of a larger set
>> >> >   # Multiple definitions are possible
>> >> >   class: [ device, spi-device ]
>> >> >
>> >> >   # This binding inherits property characteristics from the generic
>> >> >   # spi-slave binding
>> >> >   # Note that the notation is standard yaml reference
>> >> >   inherits: *spi-slave
>> >> >
>> >> >   # virtual bindings do not generate checkers
>> >> >   virtual: true
>> >>
>> >> virtual is an overloaded term.
>> >>
>> >
>> > OK, what term should I use that this binding should not be instantiated
>> > as a checker, only be used by other bindings when inherited?
>>
>> checks: true?
>>
>> I'd really like to avoid having to decide and drop this, but I don't
>> really get why it is needed.
>>
>
> It is needed because otherwise checker filters will be generated for
> the template bindings that while they won't be executed they will be
> compiled and take up space in the schema.

Size is not a problem I have. That can come later if needed.

>> >> >   # each property is defined by each name
>> >> >   properties:
>> >> >
>> >> > # The compatible property is a reserved name. The type is always
>> >> > "string"
>> >> > # and should not be repeated device binding.
>> >> > compatible:
>> >> >   category: required# required property
>> >> >   type: strseq  # is a sequence of strings
>> >> >
>> >> >   description: >
>> >> > FX11 is a clone of the original SK11 device
>> >> >
>> >> >   # v is always the 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>>  wrote:
>> > Hi Rob,
>> >
>> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>> >>  wrote:
>> >> > Hi Rob,
>> >> >
>> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >> >>  wrote:
>> >> >> > Hello again,
>> >> >> >
>> >> >> > Significant progress has been made on yamldt and is now capable of
>> >> >> > not only generating yaml from DTS source but also compiling DTS 
>> >> >> > sources
>> >> >> > and being almost fully compatible with DTC.
>> >> >>
>> >> >> Can you quantify "almost"?
>> >> >>
>> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> >> > DTC=yamldt.
>> >> >>
>> >> >> Good.
>> >> >>
>> >> >> >
>> >> >> > Error reporting is accurate and validation against a YAML based 
>> >> >> > schema
>> >> >> > works as well. In a short while I will begin posting patches with
>> >> >> > fixes on bindings and DTS files in the kernel.
>> >> >>
>> >> >> What I would like to see is the schema format posted for review.
>> >> >>
>> >> >
>> >> > I'm including the skeleton.yaml binding which is the template for
>> >> > the bindings and a board-example.yaml binding for a top level binding.
>> >> >
>> >> >> I would also like to see the bindings for top-level compatible strings
>> >> >> (aka boards) as an example. That's something that's simple enough that
>> >> >> I'd think we could agree on a format and start moving towards defining
>> >> >> board bindings that way.
>> >> >>
>> >> >
>> >> > Note there is some line wrapping I'm including a link
>> >> > to the github repo of the files:
>> >> >
>> >> >
>> >> > The skeleton.yaml
>> >> >
>> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >> >
>> >> > %YAML 1.1
>> >> > ---
>> >> > # The name of the binding is first
>> >> > # The anchor is put there for use by others
>> >> > skeleton: 
>> >>
>> >> This and "id" seem redundant.
>> >>
>> >
>> > Indeed.
>>
>> One other thing, "skeleton" is a bit weird as a key. It can't be
>> validated. All the other keys are standard words. I could write
>> "skeloton" by mistake and I guess I'd only find the mistake when
>> something inherits it. That's somewhat true with id, but we can at
>> least check "id" is present and that it's value is unique among all
>> other id values.
>>
>
> We can keep id and check that it matches the name of the enclosing node.
> That way you can be sure that there's no error.
>
> But it's a bit weird cause this is similar to declaring a function name
> with a typo. You won't find out until you use it.
>
>> >
>> >> >   version: 1
>> >> >
>> >> >   id: skel-device
>> >> >
>> >> >   title: >
>> >> > Skeleton Device
>> >> >
>> >> >   maintainer:
>> >> > name: Skeleton Person 
>> >> >
>> >> >   description: >
>> >> > The Skeleton Device binding represents the SK11 device produced by
>> >> > the Skeleton Corporation. The binding can also support compatible
>> >> > clones made by second source vendors.
>> >> >
>> >> >   # The class is an optional property that declares this
>> >> >   # binding as part of a larger set
>> >> >   # Multiple definitions are possible
>> >> >   class: [ device, spi-device ]
>> >> >
>> >> >   # This binding inherits property characteristics from the generic
>> >> >   # spi-slave binding
>> >> >   # Note that the notation is standard yaml reference
>> >> >   inherits: *spi-slave
>> >> >
>> >> >   # virtual bindings do not generate checkers
>> >> >   virtual: true
>> >>
>> >> virtual is an overloaded term.
>> >>
>> >
>> > OK, what term should I use that this binding should not be instantiated
>> > as a checker, only be used by other bindings when inherited?
>>
>> checks: true?
>>
>> I'd really like to avoid having to decide and drop this, but I don't
>> really get why it is needed.
>>
>
> It is needed because otherwise checker filters will be generated for
> the template bindings that while they won't be executed they will be
> compiled and take up space in the schema.

Size is not a problem I have. That can come later if needed.

>> >> >   # each property is defined by each name
>> >> >   properties:
>> >> >
>> >> > # The compatible property is a reserved name. The type is always
>> >> > "string"
>> >> > # and should not be repeated device binding.
>> >> > compatible:
>> >> >   category: required# required property
>> >> >   type: strseq  # is a sequence of strings
>> >> >
>> >> >   description: >
>> >> > FX11 is a clone of the original SK11 device
>> >> >
>> >> >   # v is always the name of the value of the property
>> >> >   # np is passed to the checker and is the current
>> >> >   # node pointer. We can access 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>  wrote:
> > Hi Rob,
> >
> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
> >>  wrote:
> >> > Hi Rob,
> >> >
> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >> >>  wrote:
> >> >> > Hello again,
> >> >> >
> >> >> > Significant progress has been made on yamldt and is now capable of
> >> >> > not only generating yaml from DTS source but also compiling DTS 
> >> >> > sources
> >> >> > and being almost fully compatible with DTC.
> >> >>
> >> >> Can you quantify "almost"?
> >> >>
> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> >> > DTC=yamldt.
> >> >>
> >> >> Good.
> >> >>
> >> >> >
> >> >> > Error reporting is accurate and validation against a YAML based schema
> >> >> > works as well. In a short while I will begin posting patches with
> >> >> > fixes on bindings and DTS files in the kernel.
> >> >>
> >> >> What I would like to see is the schema format posted for review.
> >> >>
> >> >
> >> > I'm including the skeleton.yaml binding which is the template for
> >> > the bindings and a board-example.yaml binding for a top level binding.
> >> >
> >> >> I would also like to see the bindings for top-level compatible strings
> >> >> (aka boards) as an example. That's something that's simple enough that
> >> >> I'd think we could agree on a format and start moving towards defining
> >> >> board bindings that way.
> >> >>
> >> >
> >> > Note there is some line wrapping I'm including a link
> >> > to the github repo of the files:
> >> >
> >> >
> >> > The skeleton.yaml
> >> >
> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >> >
> >> > %YAML 1.1
> >> > ---
> >> > # The name of the binding is first
> >> > # The anchor is put there for use by others
> >> > skeleton: 
> >>
> >> This and "id" seem redundant.
> >>
> >
> > Indeed.
> 
> One other thing, "skeleton" is a bit weird as a key. It can't be
> validated. All the other keys are standard words. I could write
> "skeloton" by mistake and I guess I'd only find the mistake when
> something inherits it. That's somewhat true with id, but we can at
> least check "id" is present and that it's value is unique among all
> other id values.
> 

We can keep id and check that it matches the name of the enclosing node.
That way you can be sure that there's no error.

But it's a bit weird cause this is similar to declaring a function name
with a typo. You won't find out until you use it.

> >
> >> >   version: 1
> >> >
> >> >   id: skel-device
> >> >
> >> >   title: >
> >> > Skeleton Device
> >> >
> >> >   maintainer:
> >> > name: Skeleton Person 
> >> >
> >> >   description: >
> >> > The Skeleton Device binding represents the SK11 device produced by
> >> > the Skeleton Corporation. The binding can also support compatible
> >> > clones made by second source vendors.
> >> >
> >> >   # The class is an optional property that declares this
> >> >   # binding as part of a larger set
> >> >   # Multiple definitions are possible
> >> >   class: [ device, spi-device ]
> >> >
> >> >   # This binding inherits property characteristics from the generic
> >> >   # spi-slave binding
> >> >   # Note that the notation is standard yaml reference
> >> >   inherits: *spi-slave
> >> >
> >> >   # virtual bindings do not generate checkers
> >> >   virtual: true
> >>
> >> virtual is an overloaded term.
> >>
> >
> > OK, what term should I use that this binding should not be instantiated
> > as a checker, only be used by other bindings when inherited?
> 
> checks: true?
> 
> I'd really like to avoid having to decide and drop this, but I don't
> really get why it is needed.
> 

It is needed because otherwise checker filters will be generated for
the template bindings that while they won't be executed they will be
compiled and take up space in the schema.

> >
> >> >
> >> >   # each property is defined by each name
> >> >   properties:
> >> >
> >> > # The compatible property is a reserved name. The type is always
> >> > "string"
> >> > # and should not be repeated device binding.
> >> > compatible:
> >> >   category: required# required property
> >> >   type: strseq  # is a sequence of strings
> >> >
> >> >   description: >
> >> > FX11 is a clone of the original SK11 device
> >> >
> >> >   # v is always the name of the value of the property
> >> >   # np is passed to the checker and is the current
> >> >   # node pointer. We can access properties and call
> >> >   # methods that operate on them.
> >> >   # There can be multiple constraints, just put them
> >> >   # into a 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote:
> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
>  wrote:
> > Hi Rob,
> >
> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
> >>  wrote:
> >> > Hi Rob,
> >> >
> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >> >>  wrote:
> >> >> > Hello again,
> >> >> >
> >> >> > Significant progress has been made on yamldt and is now capable of
> >> >> > not only generating yaml from DTS source but also compiling DTS 
> >> >> > sources
> >> >> > and being almost fully compatible with DTC.
> >> >>
> >> >> Can you quantify "almost"?
> >> >>
> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> >> > DTC=yamldt.
> >> >>
> >> >> Good.
> >> >>
> >> >> >
> >> >> > Error reporting is accurate and validation against a YAML based schema
> >> >> > works as well. In a short while I will begin posting patches with
> >> >> > fixes on bindings and DTS files in the kernel.
> >> >>
> >> >> What I would like to see is the schema format posted for review.
> >> >>
> >> >
> >> > I'm including the skeleton.yaml binding which is the template for
> >> > the bindings and a board-example.yaml binding for a top level binding.
> >> >
> >> >> I would also like to see the bindings for top-level compatible strings
> >> >> (aka boards) as an example. That's something that's simple enough that
> >> >> I'd think we could agree on a format and start moving towards defining
> >> >> board bindings that way.
> >> >>
> >> >
> >> > Note there is some line wrapping I'm including a link
> >> > to the github repo of the files:
> >> >
> >> >
> >> > The skeleton.yaml
> >> >
> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >> >
> >> > %YAML 1.1
> >> > ---
> >> > # The name of the binding is first
> >> > # The anchor is put there for use by others
> >> > skeleton: 
> >>
> >> This and "id" seem redundant.
> >>
> >
> > Indeed.
> 
> One other thing, "skeleton" is a bit weird as a key. It can't be
> validated. All the other keys are standard words. I could write
> "skeloton" by mistake and I guess I'd only find the mistake when
> something inherits it. That's somewhat true with id, but we can at
> least check "id" is present and that it's value is unique among all
> other id values.
> 

We can keep id and check that it matches the name of the enclosing node.
That way you can be sure that there's no error.

But it's a bit weird cause this is similar to declaring a function name
with a typo. You won't find out until you use it.

> >
> >> >   version: 1
> >> >
> >> >   id: skel-device
> >> >
> >> >   title: >
> >> > Skeleton Device
> >> >
> >> >   maintainer:
> >> > name: Skeleton Person 
> >> >
> >> >   description: >
> >> > The Skeleton Device binding represents the SK11 device produced by
> >> > the Skeleton Corporation. The binding can also support compatible
> >> > clones made by second source vendors.
> >> >
> >> >   # The class is an optional property that declares this
> >> >   # binding as part of a larger set
> >> >   # Multiple definitions are possible
> >> >   class: [ device, spi-device ]
> >> >
> >> >   # This binding inherits property characteristics from the generic
> >> >   # spi-slave binding
> >> >   # Note that the notation is standard yaml reference
> >> >   inherits: *spi-slave
> >> >
> >> >   # virtual bindings do not generate checkers
> >> >   virtual: true
> >>
> >> virtual is an overloaded term.
> >>
> >
> > OK, what term should I use that this binding should not be instantiated
> > as a checker, only be used by other bindings when inherited?
> 
> checks: true?
> 
> I'd really like to avoid having to decide and drop this, but I don't
> really get why it is needed.
> 

It is needed because otherwise checker filters will be generated for
the template bindings that while they won't be executed they will be
compiled and take up space in the schema.

> >
> >> >
> >> >   # each property is defined by each name
> >> >   properties:
> >> >
> >> > # The compatible property is a reserved name. The type is always
> >> > "string"
> >> > # and should not be repeated device binding.
> >> > compatible:
> >> >   category: required# required property
> >> >   type: strseq  # is a sequence of strings
> >> >
> >> >   description: >
> >> > FX11 is a clone of the original SK11 device
> >> >
> >> >   # v is always the name of the value of the property
> >> >   # np is passed to the checker and is the current
> >> >   # node pointer. We can access properties and call
> >> >   # methods that operate on them.
> >> >   # There can be multiple constraints, just put them
> >> >   # into a sequence.
> >> >   # Note that the BASE("skel,sk11") form from the previous
> >> >   # binding will have to be 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>>  wrote:
>> > Hi Rob,
>> >
>> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >>  wrote:
>> >> > Hello again,
>> >> >
>> >> > Significant progress has been made on yamldt and is now capable of
>> >> > not only generating yaml from DTS source but also compiling DTS sources
>> >> > and being almost fully compatible with DTC.
>> >>
>> >> Can you quantify "almost"?
>> >>
>> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> > DTC=yamldt.
>> >>
>> >> Good.
>> >>
>> >> >
>> >> > Error reporting is accurate and validation against a YAML based schema
>> >> > works as well. In a short while I will begin posting patches with
>> >> > fixes on bindings and DTS files in the kernel.
>> >>
>> >> What I would like to see is the schema format posted for review.
>> >>
>> >
>> > I'm including the skeleton.yaml binding which is the template for
>> > the bindings and a board-example.yaml binding for a top level binding.
>> >
>> >> I would also like to see the bindings for top-level compatible strings
>> >> (aka boards) as an example. That's something that's simple enough that
>> >> I'd think we could agree on a format and start moving towards defining
>> >> board bindings that way.
>> >>
>> >
>> > Note there is some line wrapping I'm including a link
>> > to the github repo of the files:
>> >
>> >
>> > The skeleton.yaml
>> >
>> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >
>> > %YAML 1.1
>> > ---
>> > # The name of the binding is first
>> > # The anchor is put there for use by others
>> > skeleton: 
>>
>> This and "id" seem redundant.
>>
>
> Indeed.

One other thing, "skeleton" is a bit weird as a key. It can't be
validated. All the other keys are standard words. I could write
"skeloton" by mistake and I guess I'd only find the mistake when
something inherits it. That's somewhat true with id, but we can at
least check "id" is present and that it's value is unique among all
other id values.

>
>> >   version: 1
>> >
>> >   id: skel-device
>> >
>> >   title: >
>> > Skeleton Device
>> >
>> >   maintainer:
>> > name: Skeleton Person 
>> >
>> >   description: >
>> > The Skeleton Device binding represents the SK11 device produced by
>> > the Skeleton Corporation. The binding can also support compatible
>> > clones made by second source vendors.
>> >
>> >   # The class is an optional property that declares this
>> >   # binding as part of a larger set
>> >   # Multiple definitions are possible
>> >   class: [ device, spi-device ]
>> >
>> >   # This binding inherits property characteristics from the generic
>> >   # spi-slave binding
>> >   # Note that the notation is standard yaml reference
>> >   inherits: *spi-slave
>> >
>> >   # virtual bindings do not generate checkers
>> >   virtual: true
>>
>> virtual is an overloaded term.
>>
>
> OK, what term should I use that this binding should not be instantiated
> as a checker, only be used by other bindings when inherited?

checks: true?

I'd really like to avoid having to decide and drop this, but I don't
really get why it is needed.

>
>> >
>> >   # each property is defined by each name
>> >   properties:
>> >
>> > # The compatible property is a reserved name. The type is always
>> > "string"
>> > # and should not be repeated device binding.
>> > compatible:
>> >   category: required# required property
>> >   type: strseq  # is a sequence of strings
>> >
>> >   description: >
>> > FX11 is a clone of the original SK11 device
>> >
>> >   # v is always the name of the value of the property
>> >   # np is passed to the checker and is the current
>> >   # node pointer. We can access properties and call
>> >   # methods that operate on them.
>> >   # There can be multiple constraints, just put them
>> >   # into a sequence.
>> >   # Note that the BASE("skel,sk11") form from the previous
>> >   # binding will have to be reworked.
>> >   constraint: |
>> > anystreq(v, "skel,sk11") ||
>> > anystreq(v, "faux,fx11")
>>
>> Constraints and logic ops were certainly not decided in the last
>> attempt and I think will be the hardest part to define. I see you are
>> using eBPF in the checker. Is that where anystreq comes from?
>>
>
> Yes. The ebpf environment declares a number of methods that are executed
> outside the ebpf sandbox. Check out
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
> https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

I looked at this some a while back. It wasn't clear to me 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
>> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>>  wrote:
>> > Hi Rob,
>> >
>> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>> >>  wrote:
>> >> > Hello again,
>> >> >
>> >> > Significant progress has been made on yamldt and is now capable of
>> >> > not only generating yaml from DTS source but also compiling DTS sources
>> >> > and being almost fully compatible with DTC.
>> >>
>> >> Can you quantify "almost"?
>> >>
>> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> >> > DTC=yamldt.
>> >>
>> >> Good.
>> >>
>> >> >
>> >> > Error reporting is accurate and validation against a YAML based schema
>> >> > works as well. In a short while I will begin posting patches with
>> >> > fixes on bindings and DTS files in the kernel.
>> >>
>> >> What I would like to see is the schema format posted for review.
>> >>
>> >
>> > I'm including the skeleton.yaml binding which is the template for
>> > the bindings and a board-example.yaml binding for a top level binding.
>> >
>> >> I would also like to see the bindings for top-level compatible strings
>> >> (aka boards) as an example. That's something that's simple enough that
>> >> I'd think we could agree on a format and start moving towards defining
>> >> board bindings that way.
>> >>
>> >
>> > Note there is some line wrapping I'm including a link
>> > to the github repo of the files:
>> >
>> >
>> > The skeleton.yaml
>> >
>> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>> >
>> > %YAML 1.1
>> > ---
>> > # The name of the binding is first
>> > # The anchor is put there for use by others
>> > skeleton: 
>>
>> This and "id" seem redundant.
>>
>
> Indeed.

One other thing, "skeleton" is a bit weird as a key. It can't be
validated. All the other keys are standard words. I could write
"skeloton" by mistake and I guess I'd only find the mistake when
something inherits it. That's somewhat true with id, but we can at
least check "id" is present and that it's value is unique among all
other id values.

>
>> >   version: 1
>> >
>> >   id: skel-device
>> >
>> >   title: >
>> > Skeleton Device
>> >
>> >   maintainer:
>> > name: Skeleton Person 
>> >
>> >   description: >
>> > The Skeleton Device binding represents the SK11 device produced by
>> > the Skeleton Corporation. The binding can also support compatible
>> > clones made by second source vendors.
>> >
>> >   # The class is an optional property that declares this
>> >   # binding as part of a larger set
>> >   # Multiple definitions are possible
>> >   class: [ device, spi-device ]
>> >
>> >   # This binding inherits property characteristics from the generic
>> >   # spi-slave binding
>> >   # Note that the notation is standard yaml reference
>> >   inherits: *spi-slave
>> >
>> >   # virtual bindings do not generate checkers
>> >   virtual: true
>>
>> virtual is an overloaded term.
>>
>
> OK, what term should I use that this binding should not be instantiated
> as a checker, only be used by other bindings when inherited?

checks: true?

I'd really like to avoid having to decide and drop this, but I don't
really get why it is needed.

>
>> >
>> >   # each property is defined by each name
>> >   properties:
>> >
>> > # The compatible property is a reserved name. The type is always
>> > "string"
>> > # and should not be repeated device binding.
>> > compatible:
>> >   category: required# required property
>> >   type: strseq  # is a sequence of strings
>> >
>> >   description: >
>> > FX11 is a clone of the original SK11 device
>> >
>> >   # v is always the name of the value of the property
>> >   # np is passed to the checker and is the current
>> >   # node pointer. We can access properties and call
>> >   # methods that operate on them.
>> >   # There can be multiple constraints, just put them
>> >   # into a sequence.
>> >   # Note that the BASE("skel,sk11") form from the previous
>> >   # binding will have to be reworked.
>> >   constraint: |
>> > anystreq(v, "skel,sk11") ||
>> > anystreq(v, "faux,fx11")
>>
>> Constraints and logic ops were certainly not decided in the last
>> attempt and I think will be the hardest part to define. I see you are
>> using eBPF in the checker. Is that where anystreq comes from?
>>
>
> Yes. The ebpf environment declares a number of methods that are executed
> outside the ebpf sandbox. Check out
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
> https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

I looked at this some a while back. It wasn't clear to me what eBPF
gives us and what you have defined on top of it. What I'm really after
is documentation of the syntax 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>  wrote:
> > Hi Rob,
> >
> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >>  wrote:
> >> > Hello again,
> >> >
> >> > Significant progress has been made on yamldt and is now capable of
> >> > not only generating yaml from DTS source but also compiling DTS sources
> >> > and being almost fully compatible with DTC.
> >>
> >> Can you quantify "almost"?
> >>
> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> > DTC=yamldt.
> >>
> >> Good.
> >>
> >> >
> >> > Error reporting is accurate and validation against a YAML based schema
> >> > works as well. In a short while I will begin posting patches with
> >> > fixes on bindings and DTS files in the kernel.
> >>
> >> What I would like to see is the schema format posted for review.
> >>
> >
> > I'm including the skeleton.yaml binding which is the template for
> > the bindings and a board-example.yaml binding for a top level binding.
> >
> >> I would also like to see the bindings for top-level compatible strings
> >> (aka boards) as an example. That's something that's simple enough that
> >> I'd think we could agree on a format and start moving towards defining
> >> board bindings that way.
> >>
> >
> > Note there is some line wrapping I'm including a link
> > to the github repo of the files:
> >
> >
> > The skeleton.yaml
> >
> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >
> > %YAML 1.1
> > ---
> > # The name of the binding is first
> > # The anchor is put there for use by others
> > skeleton: 
> 
> This and "id" seem redundant.
> 

Indeed.

> >   version: 1
> >
> >   id: skel-device
> >
> >   title: >
> > Skeleton Device
> >
> >   maintainer:
> > name: Skeleton Person 
> >
> >   description: >
> > The Skeleton Device binding represents the SK11 device produced by
> > the Skeleton Corporation. The binding can also support compatible
> > clones made by second source vendors.
> >
> >   # The class is an optional property that declares this
> >   # binding as part of a larger set
> >   # Multiple definitions are possible
> >   class: [ device, spi-device ]
> >
> >   # This binding inherits property characteristics from the generic
> >   # spi-slave binding
> >   # Note that the notation is standard yaml reference
> >   inherits: *spi-slave
> >
> >   # virtual bindings do not generate checkers
> >   virtual: true
> 
> virtual is an overloaded term.
> 

OK, what term should I use that this binding should not be instantiated
as a checker, only be used by other bindings when inherited?

> >
> >   # each property is defined by each name
> >   properties:
> >
> > # The compatible property is a reserved name. The type is always
> > "string"
> > # and should not be repeated device binding.
> > compatible:
> >   category: required# required property
> >   type: strseq  # is a sequence of strings
> >
> >   description: >
> > FX11 is a clone of the original SK11 device
> >
> >   # v is always the name of the value of the property
> >   # np is passed to the checker and is the current
> >   # node pointer. We can access properties and call
> >   # methods that operate on them.
> >   # There can be multiple constraints, just put them
> >   # into a sequence.
> >   # Note that the BASE("skel,sk11") form from the previous
> >   # binding will have to be reworked.
> >   constraint: |
> > anystreq(v, "skel,sk11") ||
> > anystreq(v, "faux,fx11")
> 
> Constraints and logic ops were certainly not decided in the last
> attempt and I think will be the hardest part to define. I see you are
> using eBPF in the checker. Is that where anystreq comes from?
> 

Yes. The ebpf environment declares a number of methods that are executed
outside the ebpf sandbox. Check out

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

> How would you express the ordering requirement (most significant
> compatible first)?
> 

Err, there might be new bpf API needed there. For the first stab at
the bindings problem I concentrated on getting things working and be as
clear as possible.

You could do something like that:

   orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL })

Which would check ordering too.

Obviously you'd hide the weird syntax using a macro.

#define ORDEREDSTRSEQ(_v, ...) \
orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL })


So you'd write the above as:

ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11")

> >
> > # The reg property is a reserved name. The type is always "int" and
> > # should not be repeated 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote:
> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
>  wrote:
> > Hi Rob,
> >
> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
> >>  wrote:
> >> > Hello again,
> >> >
> >> > Significant progress has been made on yamldt and is now capable of
> >> > not only generating yaml from DTS source but also compiling DTS sources
> >> > and being almost fully compatible with DTC.
> >>
> >> Can you quantify "almost"?
> >>
> >> > Compiling the kernel's DTBs using yamldt is as simple as using a
> >> > DTC=yamldt.
> >>
> >> Good.
> >>
> >> >
> >> > Error reporting is accurate and validation against a YAML based schema
> >> > works as well. In a short while I will begin posting patches with
> >> > fixes on bindings and DTS files in the kernel.
> >>
> >> What I would like to see is the schema format posted for review.
> >>
> >
> > I'm including the skeleton.yaml binding which is the template for
> > the bindings and a board-example.yaml binding for a top level binding.
> >
> >> I would also like to see the bindings for top-level compatible strings
> >> (aka boards) as an example. That's something that's simple enough that
> >> I'd think we could agree on a format and start moving towards defining
> >> board bindings that way.
> >>
> >
> > Note there is some line wrapping I'm including a link
> > to the github repo of the files:
> >
> >
> > The skeleton.yaml
> >
> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
> >
> > %YAML 1.1
> > ---
> > # The name of the binding is first
> > # The anchor is put there for use by others
> > skeleton: 
> 
> This and "id" seem redundant.
> 

Indeed.

> >   version: 1
> >
> >   id: skel-device
> >
> >   title: >
> > Skeleton Device
> >
> >   maintainer:
> > name: Skeleton Person 
> >
> >   description: >
> > The Skeleton Device binding represents the SK11 device produced by
> > the Skeleton Corporation. The binding can also support compatible
> > clones made by second source vendors.
> >
> >   # The class is an optional property that declares this
> >   # binding as part of a larger set
> >   # Multiple definitions are possible
> >   class: [ device, spi-device ]
> >
> >   # This binding inherits property characteristics from the generic
> >   # spi-slave binding
> >   # Note that the notation is standard yaml reference
> >   inherits: *spi-slave
> >
> >   # virtual bindings do not generate checkers
> >   virtual: true
> 
> virtual is an overloaded term.
> 

OK, what term should I use that this binding should not be instantiated
as a checker, only be used by other bindings when inherited?

> >
> >   # each property is defined by each name
> >   properties:
> >
> > # The compatible property is a reserved name. The type is always
> > "string"
> > # and should not be repeated device binding.
> > compatible:
> >   category: required# required property
> >   type: strseq  # is a sequence of strings
> >
> >   description: >
> > FX11 is a clone of the original SK11 device
> >
> >   # v is always the name of the value of the property
> >   # np is passed to the checker and is the current
> >   # node pointer. We can access properties and call
> >   # methods that operate on them.
> >   # There can be multiple constraints, just put them
> >   # into a sequence.
> >   # Note that the BASE("skel,sk11") form from the previous
> >   # binding will have to be reworked.
> >   constraint: |
> > anystreq(v, "skel,sk11") ||
> > anystreq(v, "faux,fx11")
> 
> Constraints and logic ops were certainly not decided in the last
> attempt and I think will be the hardest part to define. I see you are
> using eBPF in the checker. Is that where anystreq comes from?
> 

Yes. The ebpf environment declares a number of methods that are executed
outside the ebpf sandbox. Check out

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml
https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c

> How would you express the ordering requirement (most significant
> compatible first)?
> 

Err, there might be new bpf API needed there. For the first stab at
the bindings problem I concentrated on getting things working and be as
clear as possible.

You could do something like that:

   orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL })

Which would check ordering too.

Obviously you'd hide the weird syntax using a macro.

#define ORDEREDSTRSEQ(_v, ...) \
orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL })


So you'd write the above as:

ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11")

> >
> > # The reg property is a reserved name. The type is always "int" and
> > # should not be repeated in a device binding. Constraints are
> > defined
> > # only in the context of 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>>  wrote:
>> > Hello again,
>> >
>> > Significant progress has been made on yamldt and is now capable of
>> > not only generating yaml from DTS source but also compiling DTS sources
>> > and being almost fully compatible with DTC.
>>
>> Can you quantify "almost"?
>>
>> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> > DTC=yamldt.
>>
>> Good.
>>
>> >
>> > Error reporting is accurate and validation against a YAML based schema
>> > works as well. In a short while I will begin posting patches with
>> > fixes on bindings and DTS files in the kernel.
>>
>> What I would like to see is the schema format posted for review.
>>
>
> I'm including the skeleton.yaml binding which is the template for
> the bindings and a board-example.yaml binding for a top level binding.
>
>> I would also like to see the bindings for top-level compatible strings
>> (aka boards) as an example. That's something that's simple enough that
>> I'd think we could agree on a format and start moving towards defining
>> board bindings that way.
>>
>
> Note there is some line wrapping I'm including a link
> to the github repo of the files:
>
>
> The skeleton.yaml
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>
> %YAML 1.1
> ---
> # The name of the binding is first
> # The anchor is put there for use by others
> skeleton: 

This and "id" seem redundant.

>   version: 1
>
>   id: skel-device
>
>   title: >
> Skeleton Device
>
>   maintainer:
> name: Skeleton Person 
>
>   description: >
> The Skeleton Device binding represents the SK11 device produced by
> the Skeleton Corporation. The binding can also support compatible
> clones made by second source vendors.
>
>   # The class is an optional property that declares this
>   # binding as part of a larger set
>   # Multiple definitions are possible
>   class: [ device, spi-device ]
>
>   # This binding inherits property characteristics from the generic
>   # spi-slave binding
>   # Note that the notation is standard yaml reference
>   inherits: *spi-slave
>
>   # virtual bindings do not generate checkers
>   virtual: true

virtual is an overloaded term.

>
>   # each property is defined by each name
>   properties:
>
> # The compatible property is a reserved name. The type is always
> "string"
> # and should not be repeated device binding.
> compatible:
>   category: required# required property
>   type: strseq  # is a sequence of strings
>
>   description: >
> FX11 is a clone of the original SK11 device
>
>   # v is always the name of the value of the property
>   # np is passed to the checker and is the current
>   # node pointer. We can access properties and call
>   # methods that operate on them.
>   # There can be multiple constraints, just put them
>   # into a sequence.
>   # Note that the BASE("skel,sk11") form from the previous
>   # binding will have to be reworked.
>   constraint: |
> anystreq(v, "skel,sk11") ||
> anystreq(v, "faux,fx11")

Constraints and logic ops were certainly not decided in the last
attempt and I think will be the hardest part to define. I see you are
using eBPF in the checker. Is that where anystreq comes from?

How would you express the ordering requirement (most significant
compatible first)?

>
> # The reg property is a reserved name. The type is always "int" and
> # should not be repeated in a device binding. Constraints are
> defined
> # only in the context of the parent node's address, size, and ranges
> # cells. The description is inherited from the spi-slave binding.
> # Note that if inheriting from a base binding this declaration may
> # be omitted.
> reg:
>   category: required# required property
>   type: intseq  # is a sequence of integers
>
> # spi-max-frequency needs the device-specific constraint to be
> supplied
> spi-max-frequency:
>   # this constraint is dependent on the compatible property
>   # property containing "skel,sk11"
>   constraint: |
> v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
> 1000 : 100
>
> # This property overrides the generic binding description with
> # a device specific description in order to mention the chip's
> # h/w cfg strapping pins.
> spi-cs-high:
>   description: >
> Set if skeleton device configuration pins are set for chip
> select polarity high
>
> # Device specific properties don't inherit characteristic from a
> generic
> # binding so category, type, constraint, and description must be
> specified
> # if needed.
>   

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou
 wrote:
> Hi Rob,
>
> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>>  wrote:
>> > Hello again,
>> >
>> > Significant progress has been made on yamldt and is now capable of
>> > not only generating yaml from DTS source but also compiling DTS sources
>> > and being almost fully compatible with DTC.
>>
>> Can you quantify "almost"?
>>
>> > Compiling the kernel's DTBs using yamldt is as simple as using a
>> > DTC=yamldt.
>>
>> Good.
>>
>> >
>> > Error reporting is accurate and validation against a YAML based schema
>> > works as well. In a short while I will begin posting patches with
>> > fixes on bindings and DTS files in the kernel.
>>
>> What I would like to see is the schema format posted for review.
>>
>
> I'm including the skeleton.yaml binding which is the template for
> the bindings and a board-example.yaml binding for a top level binding.
>
>> I would also like to see the bindings for top-level compatible strings
>> (aka boards) as an example. That's something that's simple enough that
>> I'd think we could agree on a format and start moving towards defining
>> board bindings that way.
>>
>
> Note there is some line wrapping I'm including a link
> to the github repo of the files:
>
>
> The skeleton.yaml
>
> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml
>
> %YAML 1.1
> ---
> # The name of the binding is first
> # The anchor is put there for use by others
> skeleton: 

This and "id" seem redundant.

>   version: 1
>
>   id: skel-device
>
>   title: >
> Skeleton Device
>
>   maintainer:
> name: Skeleton Person 
>
>   description: >
> The Skeleton Device binding represents the SK11 device produced by
> the Skeleton Corporation. The binding can also support compatible
> clones made by second source vendors.
>
>   # The class is an optional property that declares this
>   # binding as part of a larger set
>   # Multiple definitions are possible
>   class: [ device, spi-device ]
>
>   # This binding inherits property characteristics from the generic
>   # spi-slave binding
>   # Note that the notation is standard yaml reference
>   inherits: *spi-slave
>
>   # virtual bindings do not generate checkers
>   virtual: true

virtual is an overloaded term.

>
>   # each property is defined by each name
>   properties:
>
> # The compatible property is a reserved name. The type is always
> "string"
> # and should not be repeated device binding.
> compatible:
>   category: required# required property
>   type: strseq  # is a sequence of strings
>
>   description: >
> FX11 is a clone of the original SK11 device
>
>   # v is always the name of the value of the property
>   # np is passed to the checker and is the current
>   # node pointer. We can access properties and call
>   # methods that operate on them.
>   # There can be multiple constraints, just put them
>   # into a sequence.
>   # Note that the BASE("skel,sk11") form from the previous
>   # binding will have to be reworked.
>   constraint: |
> anystreq(v, "skel,sk11") ||
> anystreq(v, "faux,fx11")

Constraints and logic ops were certainly not decided in the last
attempt and I think will be the hardest part to define. I see you are
using eBPF in the checker. Is that where anystreq comes from?

How would you express the ordering requirement (most significant
compatible first)?

>
> # The reg property is a reserved name. The type is always "int" and
> # should not be repeated in a device binding. Constraints are
> defined
> # only in the context of the parent node's address, size, and ranges
> # cells. The description is inherited from the spi-slave binding.
> # Note that if inheriting from a base binding this declaration may
> # be omitted.
> reg:
>   category: required# required property
>   type: intseq  # is a sequence of integers
>
> # spi-max-frequency needs the device-specific constraint to be
> supplied
> spi-max-frequency:
>   # this constraint is dependent on the compatible property
>   # property containing "skel,sk11"
>   constraint: |
> v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
> 1000 : 100
>
> # This property overrides the generic binding description with
> # a device specific description in order to mention the chip's
> # h/w cfg strapping pins.
> spi-cs-high:
>   description: >
> Set if skeleton device configuration pins are set for chip
> select polarity high
>
> # Device specific properties don't inherit characteristic from a
> generic
> # binding so category, type, constraint, and description must be
> specified
> # if needed.
> skel,deprecated1:
>   # note that the category may be declare more than 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Geert,

On Tue, 2017-10-03 at 09:17 +0200, Geert Uytterhoeven wrote:
> Hi Pantelis,
> 
> On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
>  wrote:
> > # Note that the YAML example must be validated against this binding
> > # to be an accepted entry
> 
> Indeed ;-)
> 
> >   yaml: |
> > sk11@0:
> >   compatible: "skel,sk11"
> >   reg: 0
> >   sip-max-frequency: 100
> 
> s/sip-max-frequency/spi-max-frequency/
> 

Heh, obviously the auto-check of binding examples is not working yet :)

> Gr{oetje,eeting}s,
> 
> Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- 
> ge...@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like 
> that.
> -- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Geert,

On Tue, 2017-10-03 at 09:17 +0200, Geert Uytterhoeven wrote:
> Hi Pantelis,
> 
> On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
>  wrote:
> > # Note that the YAML example must be validated against this binding
> > # to be an accepted entry
> 
> Indeed ;-)
> 
> >   yaml: |
> > sk11@0:
> >   compatible: "skel,sk11"
> >   reg: 0
> >   sip-max-frequency: 100
> 
> s/sip-max-frequency/spi-max-frequency/
> 

Heh, obviously the auto-check of binding examples is not working yet :)

> Gr{oetje,eeting}s,
> 
> Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- 
> ge...@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like 
> that.
> -- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Geert Uytterhoeven]

Hi Pantelis,

On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
 wrote:
> # Note that the YAML example must be validated against this binding
> # to be an accepted entry

Indeed ;-)

>   yaml: |
> sk11@0:
>   compatible: "skel,sk11"
>   reg: 0
>   sip-max-frequency: 100

s/sip-max-frequency/spi-max-frequency/

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Geert Uytterhoeven]

Hi Pantelis,

On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou
 wrote:
> # Note that the YAML example must be validated against this binding
> # to be an accepted entry

Indeed ;-)

>   yaml: |
> sk11@0:
>   compatible: "skel,sk11"
>   reg: 0
>   sip-max-frequency: 100

s/sip-max-frequency/spi-max-frequency/

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>  wrote:
> > Hello again,
> >
> > Significant progress has been made on yamldt and is now capable of
> > not only generating yaml from DTS source but also compiling DTS sources
> > and being almost fully compatible with DTC.
> 
> Can you quantify "almost"?
> 
> > Compiling the kernel's DTBs using yamldt is as simple as using a
> > DTC=yamldt.
> 
> Good.
> 
> >
> > Error reporting is accurate and validation against a YAML based schema
> > works as well. In a short while I will begin posting patches with
> > fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

I'm including the skeleton.yaml binding which is the template for
the bindings and a board-example.yaml binding for a top level binding.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

Note there is some line wrapping I'm including a link
to the github repo of the files:


The skeleton.yaml 

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml

%YAML 1.1
---
# The name of the binding is first
# The anchor is put there for use by others
skeleton: 

  version: 1

  id: skel-device

  title: >
Skeleton Device

  maintainer:
name: Skeleton Person 

  description: >
The Skeleton Device binding represents the SK11 device produced by
the Skeleton Corporation. The binding can also support compatible
clones made by second source vendors.

  # The class is an optional property that declares this
  # binding as part of a larger set
  # Multiple definitions are possible
  class: [ device, spi-device ]

  # This binding inherits property characteristics from the generic
  # spi-slave binding
  # Note that the notation is standard yaml reference
  inherits: *spi-slave

  # virtual bindings do not generate checkers
  virtual: true

  # each property is defined by each name
  properties:

# The compatible property is a reserved name. The type is always
"string"
# and should not be repeated device binding.
compatible:
  category: required# required property
  type: strseq  # is a sequence of strings

  description: >
FX11 is a clone of the original SK11 device

  # v is always the name of the value of the property
  # np is passed to the checker and is the current
  # node pointer. We can access properties and call
  # methods that operate on them.
  # There can be multiple constraints, just put them
  # into a sequence.
  # Note that the BASE("skel,sk11") form from the previous
  # binding will have to be reworked.
  constraint: |
anystreq(v, "skel,sk11") ||
anystreq(v, "faux,fx11")

# The reg property is a reserved name. The type is always "int" and
# should not be repeated in a device binding. Constraints are
defined
# only in the context of the parent node's address, size, and ranges
# cells. The description is inherited from the spi-slave binding.
# Note that if inheriting from a base binding this declaration may
# be omitted.
reg:
  category: required# required property
  type: intseq  # is a sequence of integers

# spi-max-frequency needs the device-specific constraint to be
supplied
spi-max-frequency:
  # this constraint is dependent on the compatible property
  # property containing "skel,sk11"
  constraint: |
v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
1000 : 100

# This property overrides the generic binding description with
# a device specific description in order to mention the chip's
# h/w cfg strapping pins.
spi-cs-high:
  description: >
Set if skeleton device configuration pins are set for chip
select polarity high

# Device specific properties don't inherit characteristic from a
generic
# binding so category, type, constraint, and description must be
specified
# if needed.
skel,deprecated1:
  # note that the category may be declare more than one option
  category: [ deprecated, optional ]
  type: int
  constraint: |
v >= 10 && v <= 20
  description: >
First of two deprecated properties.

# There are no constraints for properties of empty type
skel,deprecated2:
  category: deprecated
  type: empty
  description: >
Second of two deprecated properties.

# This example could be auto-generated rather than explicitly
included
# in the yaml source.
# Note that the YAML example must be validated against this binding
# to be an 

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hi Rob,

On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote:
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>  wrote:
> > Hello again,
> >
> > Significant progress has been made on yamldt and is now capable of
> > not only generating yaml from DTS source but also compiling DTS sources
> > and being almost fully compatible with DTC.
> 
> Can you quantify "almost"?
> 
> > Compiling the kernel's DTBs using yamldt is as simple as using a
> > DTC=yamldt.
> 
> Good.
> 
> >
> > Error reporting is accurate and validation against a YAML based schema
> > works as well. In a short while I will begin posting patches with
> > fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

I'm including the skeleton.yaml binding which is the template for
the bindings and a board-example.yaml binding for a top level binding.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

Note there is some line wrapping I'm including a link
to the github repo of the files:


The skeleton.yaml 

https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml

%YAML 1.1
---
# The name of the binding is first
# The anchor is put there for use by others
skeleton: 

  version: 1

  id: skel-device

  title: >
Skeleton Device

  maintainer:
name: Skeleton Person 

  description: >
The Skeleton Device binding represents the SK11 device produced by
the Skeleton Corporation. The binding can also support compatible
clones made by second source vendors.

  # The class is an optional property that declares this
  # binding as part of a larger set
  # Multiple definitions are possible
  class: [ device, spi-device ]

  # This binding inherits property characteristics from the generic
  # spi-slave binding
  # Note that the notation is standard yaml reference
  inherits: *spi-slave

  # virtual bindings do not generate checkers
  virtual: true

  # each property is defined by each name
  properties:

# The compatible property is a reserved name. The type is always
"string"
# and should not be repeated device binding.
compatible:
  category: required# required property
  type: strseq  # is a sequence of strings

  description: >
FX11 is a clone of the original SK11 device

  # v is always the name of the value of the property
  # np is passed to the checker and is the current
  # node pointer. We can access properties and call
  # methods that operate on them.
  # There can be multiple constraints, just put them
  # into a sequence.
  # Note that the BASE("skel,sk11") form from the previous
  # binding will have to be reworked.
  constraint: |
anystreq(v, "skel,sk11") ||
anystreq(v, "faux,fx11")

# The reg property is a reserved name. The type is always "int" and
# should not be repeated in a device binding. Constraints are
defined
# only in the context of the parent node's address, size, and ranges
# cells. The description is inherited from the spi-slave binding.
# Note that if inheriting from a base binding this declaration may
# be omitted.
reg:
  category: required# required property
  type: intseq  # is a sequence of integers

# spi-max-frequency needs the device-specific constraint to be
supplied
spi-max-frequency:
  # this constraint is dependent on the compatible property
  # property containing "skel,sk11"
  constraint: |
v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ?
1000 : 100

# This property overrides the generic binding description with
# a device specific description in order to mention the chip's
# h/w cfg strapping pins.
spi-cs-high:
  description: >
Set if skeleton device configuration pins are set for chip
select polarity high

# Device specific properties don't inherit characteristic from a
generic
# binding so category, type, constraint, and description must be
specified
# if needed.
skel,deprecated1:
  # note that the category may be declare more than one option
  category: [ deprecated, optional ]
  type: int
  constraint: |
v >= 10 && v <= 20
  description: >
First of two deprecated properties.

# There are no constraints for properties of empty type
skel,deprecated2:
  category: deprecated
  type: empty
  description: >
Second of two deprecated properties.

# This example could be auto-generated rather than explicitly
included
# in the yaml source.
# Note that the YAML example must be validated against this binding
# to be an accepted entry
example:

  dts: |

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

> On Oct 2, 2017, at 01:00 , Rob Herring  wrote:
> 
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>  wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
> 
> Can you quantify "almost”?

The —sort option and /include-bin/ doesn’t work yet.

It’s a day or so to add them but there are no kernel dtb files that rely on 
them.

> 

>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
> 
> Good.
> 
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

It is based on your binding schema you’ve posted but tweaked slightly to
be parseable by the YAML subset that matches the DTS style.

In particular bare sequences do not map to DT at all, so:

- name: foo

is changed to
name: foo

And so on.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

I’ll see what I can do.


> Rob

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

> On Oct 2, 2017, at 01:00 , Rob Herring  wrote:
> 
> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
>  wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
> 
> Can you quantify "almost”?

The —sort option and /include-bin/ doesn’t work yet.

It’s a day or so to add them but there are no kernel dtb files that rely on 
them.

> 

>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
> 
> Good.
> 
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
> 
> What I would like to see is the schema format posted for review.
> 

It is based on your binding schema you’ve posted but tweaked slightly to
be parseable by the YAML subset that matches the DTS style.

In particular bare sequences do not map to DT at all, so:

- name: foo

is changed to
name: foo

And so on.

> I would also like to see the bindings for top-level compatible strings
> (aka boards) as an example. That's something that's simple enough that
> I'd think we could agree on a format and start moving towards defining
> board bindings that way.
> 

I’ll see what I can do.


> Rob

Regards

— Pantelis

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
 wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.

Can you quantify "almost"?

> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.

Good.

>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.

What I would like to see is the schema format posted for review.

I would also like to see the bindings for top-level compatible strings
(aka boards) as an example. That's something that's simple enough that
I'd think we could agree on a format and start moving towards defining
board bindings that way.

Rob

Re: [RFC] yamldt v0.5, now a DTS compiler too

[Rob Herring]

On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou
 wrote:
> Hello again,
>
> Significant progress has been made on yamldt and is now capable of
> not only generating yaml from DTS source but also compiling DTS sources
> and being almost fully compatible with DTC.

Can you quantify "almost"?

> Compiling the kernel's DTBs using yamldt is as simple as using a
> DTC=yamldt.

Good.

>
> Error reporting is accurate and validation against a YAML based schema
> works as well. In a short while I will begin posting patches with
> fixes on bindings and DTS files in the kernel.

What I would like to see is the schema format posted for review.

I would also like to see the bindings for top-level compatible strings
(aka boards) as an example. That's something that's simple enough that
I'd think we could agree on a format and start moving towards defining
board bindings that way.

Rob

[RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hello again,

Significant progress has been made on yamldt and is now capable of
not only generating yaml from DTS source but also compiling DTS sources
and being almost fully compatible with DTC.

Compiling the kernel's DTBs using yamldt is as simple as using a
DTC=yamldt.

Error reporting is accurate and validation against a YAML based schema
works as well. In a short while I will begin posting patches with
fixes on bindings and DTS files in the kernel.

Please try it on your platform and report if you encounter any problems.

https://github.com/pantoniou/yamldt

I am eagerly awaiting for your comments.

Regards

-- Pantelis

[RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou]

Hello again,

Significant progress has been made on yamldt and is now capable of
not only generating yaml from DTS source but also compiling DTS sources
and being almost fully compatible with DTC.

Compiling the kernel's DTBs using yamldt is as simple as using a
DTC=yamldt.

Error reporting is accurate and validation against a YAML based schema
works as well. In a short while I will begin posting patches with
fixes on bindings and DTS files in the kernel.

Please try it on your platform and report if you encounter any problems.

https://github.com/pantoniou/yamldt

I am eagerly awaiting for your comments.

Regards

-- Pantelis