Hi Lada,
I agree with your statement that YANG is designed to model datastore and
the protocol messages.
Right now the question I am trying to answer is this scenario: we have the
protocols (hence the protocol messages) first, and would like to know
whether we would be able to use YANG to model the content messages (in
order to use the YANG tools for this protocol, etc.). And the question is
what are the conditions for JSON msgs to be able to be modeled by YANG.
For JSON arrays:
0. a standalone non-object JSON value is not model-able in YANG:
examples:
--"foo"
--14
--[1,2,3]
which translates to the following in XML:
<0>1</0>
<1>2</1>
<2>3</2>
This is why your example "foo": [ { "bar": 1 }, 2 ] cannot be
modeled--because 2 is a standalone value and {"bar": 1} is an object,
otherwise there are following ways to model an array with a mix of elements:
1. an array with only built-in types or derived types (standalone JSON
values except for arrays and objects) can be modeled in YANG as the
following:
"foo" : [1, "abc", enum1]
leaf-list foo {
type union {
type int64;
type string;
type enumeration {
enum enum1;
}
}
}
2. an array of all objects
(1) This structure is model-able as well, right?
"foo": [{"a": 1}, {"a": 1, "b": 2}]
list foo {
leaf a {
type int;
}
leaf b {
type int;
}
}
(2) how about this?
"foo": [{"a": 1, "b": 2, "c": 3}, {"c": 1, "b":3, "y": 2}]
If the JSON-XML translate it to the following:
<foo>
<a>1</a>
<b>2</b>
<c>3</c>
</foo>
<foo>
<c>1</c>
<b>3</b>
<y>2</y>
</foo>
We can still model this using the choice statement, right?
If the above statements about whether these JSON texts are model-able are
correct, essentially the only JSON arrays that are not model-able are (a)
those mixed with objects and a standalone non-object JSON value; and (b)
those whose elements is an array. And this gap is caused by the difference
between leaf-list modeling capabilities of handling arrays with non-object,
non-array, standalone JSON values, and the list+choice modeling
capabilities of modeling arrays with objects. Does this sound plausible?
Cheers,
Xiao
On Mon, Oct 20, 2014 at 11:56 AM, Ladislav Lhotka <[email protected]> wrote:
> Xiao SHI <[email protected]> writes:
>
> > Hi Lada,
> >
> > Thank you so much for your feedback!
> >
> > The motivation here is to produce a YANG model given a defined protocol,
> > e.g. ALTO, CDNi, etc. This way, we may incorporate future protocols into
> > ODL controllers or use other YANG related tools to handle those
> protocols.
> > Hence the objective of the modeling is the protocol, not the messages. I
> am
> > not suggesting to use YANG as a schema language.
>
> Hmm, YANG was designed for modelling datastores and contents of protocol
> messages, not protocols as such.
>
> >
> > The conditions discussed in this process will be helpful in the following
> > sense:
> > 1) disprove the possibility to get a syntactically equivalent YANG model
> > from a JSON based protocol;
> > 2) generate a Jackson JSON parser class and YANG model from the protocols
> > that meet the criteria;
> > 3) for designers of future protocols who are thinking of using YANG
> related
> > tools, set a guideline of what their messages should look like (or MUST
> not
> > contain);
> > 4) for the YANG/NETCONF WG, describing the compatibility issues that they
> > might be interested in tackling for expanding YANG's usage.
> >
> > For this example, "foo": [ { "bar": 1 }, 2 ] is also not easy to do
> > data-binding in the parser, because the elements of the array are not of
> > the same type, which is indeed an assumption that I left out.
> >
> > For the {"foo”:null} example, can't we model it using type empty? That's
> > equivalent to the XML element <foo /> right?
>
> <foo/> corresponds to
>
> "foo": [null]
>
> Section 6.9 in draft-ietf-netmod-yang-json-01 gives the reasoning,
> essentially it is because some software isn't able to distinguish
>
> "foo": null
>
> from "foo" being absent.
>
> Lada
>
> >
> > Cheers,
> > Xiao
> >
> > On Mon, Oct 20, 2014 at 9:23 AM, Ladislav Lhotka <[email protected]> wrote:
> >
> >> Xiao SHI <[email protected]> writes:
> >>
> >> > Hi folks,
> >> >
> >> >
> >> > As a few of us were working on modeling the ALTO protocol using YANG,
> we
> >> > were pondering on a more general question: can YANG model all JSON
> based
> >> > protocols? What is the condition for a JSON based protocol (at least
> the
> >> > message format) to have an syntactically equivalent, hence
> interoperable,
> >> > YANG model with JSON encoding? Alternatively, in order to
> interoperate,
> >> > semantic equivalence might be sufficient, is there any condition for
> >> > semantic equivalence?
> >> >
> >> >
> >> > tl;dr: A JSON based protocol carried by HTTP can have syntactically
> >> > equivalent YANG model, if and only if all the keys in the key-value
> pairs
> >> > in the JSON message are pre-defined keywords in the protocol.
> >> >
> >>
> >> Generating a YANG data model from existing instance data seems backwards
> >> to me, although Examplotron (http://examplotron.org) does something
> quite
> >> similar. In any case, it might be quite challenging because YANG isn't
> >> intended as a general schema language. Your observation above is true
> >> but it is just one aspect of the problem. For instance
> >>
> >> "foo": [ { "bar": 1 }, 2 ]
> >>
> >> or
> >>
> >> "foo": null
> >>
> >> cannot be modelled in YANG using the encoding of
> >> draft-ietf-netmod-yang-json.
> >>
> >> Lada
> >>
> >> >
> >> > We've come up with a few ideas, and a few things below are work in
> >> > progress, but we would love your feedback!
> >> >
> >> >
> >> > Thank you!
> >> >
> >> > Xiao
> >> >
> >> >
> >> > ======================
> >> >
> >> > 1. Introduction.
> >> >
> >> > JavaScript Object Notation (JSON) has been a popular choice as the
> >> message
> >> > encoding for many network protocols such as the Application Layer
> Traffic
> >> > Optimization (ALTO) protocol, the Content Delivery Networks
> >> Interconnection
> >> > (CDNi) protocol, etc.
> >> >
> >> > Meanwhile, there are broad interests in the networking community to
> use
> >> > YANG to define data model so that one can use YANG related tools such
> as
> >> > OpenDayLight controller. Although YANG itself is XML based, there have
> >> been
> >> > efforts to model JSON content using YANG
> >> [draft-ietf-netmod-yang-json-01].
> >> >
> >> > A natural question rises: can YANG model all JSON based protocols?
> What
> >> is
> >> > the condition for a JSON based protocol (at least the message format)
> to
> >> > have an syntactically equivalent, hence interoperable, YANG model with
> >> JSON
> >> > encoding? Alternatively, in order to interoperate, semantic
> equivalence
> >> > might be sufficient, is there any condition for semantic equivalence?
> >> >
> >> > 2. Claim
> >> >
> >> > A JSON based protocol carried by HTTP can have syntactically
> equivalent
> >> > YANG model, if and only if:
> >> >
> >> > (1) the message encoding condition is met;
> >> >
> >> > (2) the uri condition is met.
> >> >
> >> > 2.1. The message encoding condition
> >> >
> >> > The JSON message encoding MUST not contain a variable as a key in a
> JSON
> >> > object key-value pair. In other words, the keys in the JSON message
> must
> >> be
> >> > an already defined constant (or keyword) in the message format of the
> >> > protocol.
> >> >
> >> > For example, if I have a protocol where “network-map”, “src”, and
> “dsts”
> >> > are defined keywords, JSON text A meets the condition whereas B does
> not
> >> > because “PID1” and “PID2” are not protocol keywords (they are resource
> >> IDs).
> >> >
> >> > A:
> >> >
> >> > {
> >> >
> >> > “network-map”: [
> >> >
> >> > {
> >> >
> >> > “src”: “PID1”,
> >> >
> >> > “dsts”: [“PID2”, “PID3”]
> >> >
> >> > },
> >> >
> >> > {
> >> >
> >> > “src”: “PID2”,
> >> >
> >> > “dsts”: [“PID3”, “PID4”]
> >> >
> >> > }
> >> >
> >> > ]
> >> >
> >> > }
> >> >
> >> > B:
> >> >
> >> >
> >> > {
> >> >
> >> > “network-map”: [
> >> >
> >> > “PID1: [“PID2”, “PID3”],
> >> >
> >> > “PID2”: [“PID3”, “PID4”]
> >> >
> >> > ]
> >> >
> >> > }
> >> >
> >> >>From this we know that since ALTO protocol uses encoding B, there
> cannot
> >> be
> >> > a syntactically equivalent YANG model.
> >> >
> >> > 2.2 The URI condition
> >> >
> >> > Some of the YANG-related protocols might have URI constraints, e.g.
> >> > RESTCONF. For now, we assume either that the JSON-based protocol URI
> >> could
> >> > be conformed to RESTCONF compliant uri, or that the server could have
> a
> >> > routing mapping between the protocol compliant uri and the RESTCONF
> >> > compliant uri, hence this condition would not be an issue, which
> allows
> >> us
> >> > to focus on the message encoding condition.
> >> >
> >> > 3. Proof
> >> >
> >> > 3.1. The message encoding condition is necessary.
> >> >
> >> > We first note that this condition is a necessary condition for a JSON
> >> based
> >> > protocol to have a syntactically equivalent YANG model by proving its
> >> > contrapositive.
> >> >
> >> > If one of the keys in the key-value pair in the JSON document is not
> >> > pre-defined, the corresponding XML tags will not be pre-defined
> keywords.
> >> > Therefore, it would not possible to model it in YANG without using
> >> > YANG’s anyxml
> >> > statement (which allows arbitrary XML content).
> >> >
> >> > However, using the anyxml statement would defeat our purpose of
> modeling
> >> > the data as it allows arbitrary XML content, and will not be helpful
> in
> >> the
> >> > subsequent parsing process.
> >> >
> >> > 3.2. The message encoding condition is sufficient.
> >> >
> >> > We prove this by providing a translation procedure from a JSON message
> >> that
> >> > is compliant with the protocol we are trying to model, to a custom
> java
> >> > class that can be used for Jackson data binding, then to a YANG model.
> >> >
> >> > We note that the middle step of translating to and from the custom
> parser
> >> > class is not necessary, but it will be useful.
> >> >
> >> > 3.2.1. motivation
> >> >
> >> > JSON data binding is the process of binding data structures (objects,
> >> > arrays, etc.) in JSON to the appropriate data structures in the server
> >> > (e.g. java classes, database tables, etc.) in parsing the JSON text.
> In
> >> > order to process JSON messages in a meaningful manner, data binding is
> >> > necessary. Even if the binding is not explicit, the server would need
> to
> >> do
> >> > it eventually. For example, one can read JSON content in a stream
> without
> >> > binding it to the java classes, but eventually in order to make sense
> of
> >> > the data, the server would eventually have to organize it, which is
> >> roughly
> >> > analogous to data binding upfront. Popular choices for JSON parsing
> and
> >> > data binding include jackson and gson.
> >> >
> >> > We use Jackson full data binding as our approach. Full data binding
> binds
> >> > JSON content into plain old java objects (POJOs), i.e. this custom
> parser
> >> > class can neither extend nor implement any other class. Jackson uses
> >> > ObjectMapper with the custom parser class to parse JSON content into
> this
> >> > class.
> >> >
> >> > 3.2.2. The message encoding condition
> >> >
> >> > The message encoding condition is that all keys in each key-value
> pair in
> >> > the JSON text must be pre-defined keywords. As the keys will become
> >> either
> >> > class names and instance variable names, or be keys in the java maps,
> it
> >> is
> >> > easy to see that the condition is equivalent to “there exists a full
> data
> >> > binding in Jackson custom parser class without using any map
> structures
> >> > (Map<String, ?>).”
> >> >
> >> > 3.2.3. Proof
> >> >
> >> > We provide a recursive binding process from a JSON object to the
> Jackson
> >> > custom parser class to be used by Jackson ObjectMapper.
> >> >
> >> > Type determine_type(value) {
> >> >
> >> > if (value is of a primitive type, i.e. string, number, boolean,
> null) {
> >> >
> >> > return the corresponding java primitive type;
> >> >
> >> > }
> >> >
> >> > if (value is a JSON object) {
> >> >
> >> > return build_parser_class(value).class;
> >> >
> >> > }
> >> >
> >> > if (value is an array) {
> >> >
> >> > return ArrayList<T> where T=determine_type(value[0]);
> >> >
> >> > }
> >> >
> >> > // should not reach here.
> >> >
> >> > }
> >> >
> >> > Class build_parser_class(JSONObject obj) {
> >> >
> >> > create custom class C;
> >> >
> >> > for each key/value pair in the obj {
> >> >
> >> > add instance variable v in C;
> >> >
> >> > the name of variable v <- key; // (__known__ a/c to our assumption)
> >> >
> >> > the type of variable v <- determine_type(value);
> >> >
> >> > }
> >> >
> >> > return C.class;
> >> >
> >> > }
> >> >
> >> > Naming:
> >> >
> >> > --change everything into CamelCase (i.e. remove dashes, etc.)
> >> >
> >> > --for instance variables, use “my” prefix, (e.g. myVariable,
> >> myNetworkMap,
> >> > etc.)
> >> >
> >> > --for the custom class name, if the object is an element of the array,
> >> use
> >> > “Element” suffix.
> >> >
> >> > This is just one convention so that the next step proceeds smoothly.
> As
> >> > long as this naming translation is consistent with the naming stage in
> >> the
> >> > next step, it will work just fine.
> >> >
> >> > Example (a modified ALTO protocol network map example):
> >> >
> >> > JSON object:
> >> >
> >> > {
> >> >
> >> > “meta”: {
> >> >
> >> > “resource-id”: “my-default-map”,
> >> >
> >> > “tag”: “aab875ef69c87d012”
> >> >
> >> > },
> >> >
> >> > “network-map”: [
> >> >
> >> > {
> >> >
> >> > “src”: “PID1”,
> >> >
> >> > “dsts”: [“PID1”, “PID2”, “PID3”]
> >> >
> >> > },
> >> >
> >> > {
> >> >
> >> > “src”: “PID2”,
> >> >
> >> > “dsts”: [“PID1”, “PID3”]
> >> >
> >> > },
> >> >
> >> > {
> >> >
> >> > “src”: “PID3”,
> >> >
> >> > “dsts”: [“PID2”, “PID3”]
> >> >
> >> > }
> >> >
> >> > ]
> >> >
> >> > }
> >> >
> >> > Result of build_parser_class(obj):
> >> >
> >> > Class JSONObject {
> >> >
> >> > Meta myMeta;
> >> >
> >> > ArrayList<NetworkMapElement> myNetworkMap;
> >> >
> >> > }
> >> >
> >> > Class Meta {
> >> >
> >> > String myResourceId;
> >> >
> >> > String myTag;
> >> >
> >> > }
> >> >
> >> > Class NetworkMapElement {
> >> >
> >> > String mySrc;
> >> >
> >> > ArrayList<String> myDsts;
> >> >
> >> > }
> >> >
> >> > Now given the Jackson Parser Java Class, to get a syntactically
> >> equivalent
> >> > YANG model:
> >> >
> >> > YANGModel build_yang_model(Class C) {
> >> >
> >> > for each instance variable (Type, Name) {
> >> >
> >> > if (Type is primitive type: string, number, boolean, null) {
> >> >
> >> > add the following to the YANG module:
> >> >
> >> > “leaf Name { type <YANG equivalent of Type>; }”
> >> >
> >> > }
> >> >
> >> > if (Type is an ArrayList<TypeElement>) {
> >> >
> >> > if (TypeElement is primitive type) {
> >> >
> >> > add the following to the YANG module:
> >> >
> >> > “leaf-list Name { type <YANG equivalent of TypeElement>; }”
> >> >
> >> > } else {
> >> >
> >> > // TypeElement is a custom parser class
> >> >
> >> > add the following to the YANG module:
> >> >
> >> > “list Name { <result from build_yang_model<TypeElement.class>>
> }”
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > if (Type is a custom parser class) {
> >> >
> >> > add the following to the YANG module:
> >> >
> >> > “container Name { <result from build_yang_model<Type.class>> }”
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > Result from the previous example:
> >> >
> >> > container meta {
> >> >
> >> > leaf resource-id {
> >> >
> >> > type string;
> >> >
> >> > }
> >> >
> >> > leaf tag {
> >> >
> >> > type string;
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > list network-map {
> >> >
> >> > leaf src {
> >> >
> >> > type string;
> >> >
> >> > }
> >> >
> >> > leaf-list dsts {
> >> >
> >> > type string;
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > This does validate the JSON document with XML-JSON encoding. For your
> >> > reference, this is the XML document which validates:
> >> >
> >> > <?xml version="1.0" encoding="UTF-8" ?>
> >> >
> >> > <meta>
> >> >
> >> > <resource-id>my-default-map</resource-id>
> >> >
> >> > <tag>aab875ef69c87d012</tag>
> >> >
> >> > </meta>
> >> >
> >> > <network-map>
> >> >
> >> > <src>PID1</src>
> >> >
> >> > <dsts>PID1</dsts>
> >> >
> >> > <dsts>PID2</dsts>
> >> >
> >> > <dsts>PID3</dsts>
> >> >
> >> > </network-map>
> >> >
> >> > <network-map>
> >> >
> >> > <src>PID2</src>
> >> >
> >> > <dsts>PID1</dsts>
> >> >
> >> > <dsts>PID3</dsts>
> >> >
> >> > </network-map>
> >> >
> >> > <network-map>
> >> >
> >> > <src>PID3</src>
> >> >
> >> > <dsts>PID2</dsts>
> >> >
> >> > <dsts>PID3</dsts>
> >> >
> >> > </network-map>
> >> >
> >> > This proves that the message encoding condition is a sufficient
> condition
> >> > for the JSON object to have a YANG model.
> >> >
> >> > Note the model generated is very crude and lose almost all constraints
> >> and
> >> > all inheritance features (if any), because it focuses on the syntax
> and
> >> is
> >> > essentially converted from an JSON object compliant with a protocol
> >> instead
> >> > of from the protocol itself. Hence this result is more useful in
> >> > determining which JSON based protocols cannot have a syntactically
> >> > equivalent YANG model, than in generating a good YANG model.
> >> >
> >> > 3.3. Conclusion
> >> >
> >> > Our claim holds. A JSON based protocol carried by HTTP can have
> >> > syntactically equivalent YANG model, if and only if all the keys in
> the
> >> > key-value pairs in the JSON message are pre-defined keywords.
> >> >
> >> > 4. Semantic equivalence
> >> >
> >> > For JSON based protocols that don’t satisfy the message encoding
> >> condition,
> >> > it is still possible to have a semantically equivalent YANG model. All
> >> that
> >> > is required for the protocol compliant clients and the YANG model
> >> compliant
> >> > server to interoperate is an adapter which does the following:
> >> >
> >> > 1) translate FROM YANG server compliant response msg TO alto compliant
> >> > response msg
> >> >
> >> > 2) translate FROM alto compliant request msg TO YANG server compliant
> >> > request msg
> >> >
> >> > 4.1. Claim
> >> >
> >> > This adapter needs to be protocol-aware.
> >> >
> >> > Ideally, given any YANG model, we would like to be able to
> automatically
> >> > (or at least mechanically) generate this message adapter, which means
> not
> >> > looking at the protocol or its compliant msgs. However, without
> knowing
> >> the
> >> > specific protocol that we are working with (i.e. human intervention,
> i.e.
> >> > looking at the protocol compliant msgs), such an adapter cannot be
> >> > auto-generated.
> >> >
> >> > 4.2. Proof by Indistinguishability
> >> >
> >> > Suppose both the YANG server compliant msg m_y and the actually
> protocol
> >> > compliant msg m_p are in JSON (or have been encoded into JSON).
> Looking
> >> at
> >> > the differences between the two messages, call these differences {d1,
> d2,
> >> > ..., dn}. The goal for the auto-generated adapter would be to identify
> >> and
> >> > eliminate these differences. Construct a new JSON msg m' where all but
> >> one
> >> > difference di is the same as m_p and di is the same as the m_y.
> Without
> >> > looking at the protocol (or m_p), the auto-generated adapter would
> not be
> >> > able to distinguish between m' and m_p in its translation process,
> which
> >> > means, it won't be able to tell whether it should change di or not.
> >> Hence,
> >> > such an adapter must be protocol-aware.
> >> >
> >> > A good example is the dependent-vtag in the ALTO protocol:
> >> >
> >> > "dependent-vtag" : [
> >> >
> >> > {
> >> >
> >> > "resource-id" : "my-network-map",
> >> >
> >> > "tag" : "abcd1234"
> >> >
> >> > }
> >> >
> >> > ]
> >> >
> >> > It was specified this way in the alto protocol. However, it could
> >> > conceivably be the case that it was originally the following map
> >> structure,
> >> > and was converted into the above encoding because of the map->list+key
> >> > issue. (This case is actually one of the few differences in the m_y
> and
> >> m_p
> >> > where the adapter does not need to convert it back to a map
> structure.)
> >> >
> >> > "dependent-vtag" : {
> >> >
> >> > "my-network-map" : {
> >> >
> >> > "tag" : "abcd1234"
> >> >
> >> > }
> >> >
> >> > }
> >> >
> >> > Without knowing the protocol, there is no way to tell.
> >> >
> >> > 5. Ramifications
> >> >
> >> > We now understand the basic condition for a JSON based protocol to
> have a
> >> > YANG Model. For the protocols that don’t meet this condition, there
> can
> >> be
> >> > a semantic equivalent YANG model, but there won’t be a generic
> process of
> >> > generating the adapter for all such protocols.
> >> > _______________________________________________
> >> > netmod mailing list
> >> > [email protected]
> >> > https://www.ietf.org/mailman/listinfo/netmod
> >>
> >> --
> >> Ladislav Lhotka, CZ.NIC Labs
> >> PGP Key ID: E74E8C0C
> >>
>
> --
> Ladislav Lhotka, CZ.NIC Labs
> PGP Key ID: E74E8C0C
>
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
