Hi Lada,
Thank you so much for your feedback!
The motivation here is to produce a YANG model given a defined protocol,
e.g. ALTO, CDNi, etc. This way, we may incorporate future protocols into
ODL controllers or use other YANG related tools to handle those protocols.
Hence the objective of the modeling is the protocol, not the messages. I am
not suggesting to use YANG as a schema language.
The conditions discussed in this process will be helpful in the following
sense:
1) disprove the possibility to get a syntactically equivalent YANG model
from a JSON based protocol;
2) generate a Jackson JSON parser class and YANG model from the protocols
that meet the criteria;
3) for designers of future protocols who are thinking of using YANG related
tools, set a guideline of what their messages should look like (or MUST not
contain);
4) for the YANG/NETCONF WG, describing the compatibility issues that they
might be interested in tackling for expanding YANG's usage.
For this example, "foo": [ { "bar": 1 }, 2 ] is also not easy to do
data-binding in the parser, because the elements of the array are not of
the same type, which is indeed an assumption that I left out.
For the {"foo”:null} example, can't we model it using type empty? That's
equivalent to the XML element <foo /> right?
Cheers,
Xiao
On Mon, Oct 20, 2014 at 9:23 AM, Ladislav Lhotka <[email protected]> wrote:
> Xiao SHI <[email protected]> writes:
>
> > Hi folks,
> >
> >
> > As a few of us were working on modeling the ALTO protocol using YANG, we
> > were pondering on a more general question: can YANG model all JSON based
> > protocols? What is the condition for a JSON based protocol (at least the
> > message format) to have an syntactically equivalent, hence interoperable,
> > YANG model with JSON encoding? Alternatively, in order to interoperate,
> > semantic equivalence might be sufficient, is there any condition for
> > semantic equivalence?
> >
> >
> > tl;dr: A JSON based protocol carried by HTTP can have syntactically
> > equivalent YANG model, if and only if all the keys in the key-value pairs
> > in the JSON message are pre-defined keywords in the protocol.
> >
>
> Generating a YANG data model from existing instance data seems backwards
> to me, although Examplotron (http://examplotron.org) does something quite
> similar. In any case, it might be quite challenging because YANG isn't
> intended as a general schema language. Your observation above is true
> but it is just one aspect of the problem. For instance
>
> "foo": [ { "bar": 1 }, 2 ]
>
> or
>
> "foo": null
>
> cannot be modelled in YANG using the encoding of
> draft-ietf-netmod-yang-json.
>
> Lada
>
> >
> > We've come up with a few ideas, and a few things below are work in
> > progress, but we would love your feedback!
> >
> >
> > Thank you!
> >
> > Xiao
> >
> >
> > ======================
> >
> > 1. Introduction.
> >
> > JavaScript Object Notation (JSON) has been a popular choice as the
> message
> > encoding for many network protocols such as the Application Layer Traffic
> > Optimization (ALTO) protocol, the Content Delivery Networks
> Interconnection
> > (CDNi) protocol, etc.
> >
> > Meanwhile, there are broad interests in the networking community to use
> > YANG to define data model so that one can use YANG related tools such as
> > OpenDayLight controller. Although YANG itself is XML based, there have
> been
> > efforts to model JSON content using YANG
> [draft-ietf-netmod-yang-json-01].
> >
> > A natural question rises: can YANG model all JSON based protocols? What
> is
> > the condition for a JSON based protocol (at least the message format) to
> > have an syntactically equivalent, hence interoperable, YANG model with
> JSON
> > encoding? Alternatively, in order to interoperate, semantic equivalence
> > might be sufficient, is there any condition for semantic equivalence?
> >
> > 2. Claim
> >
> > A JSON based protocol carried by HTTP can have syntactically equivalent
> > YANG model, if and only if:
> >
> > (1) the message encoding condition is met;
> >
> > (2) the uri condition is met.
> >
> > 2.1. The message encoding condition
> >
> > The JSON message encoding MUST not contain a variable as a key in a JSON
> > object key-value pair. In other words, the keys in the JSON message must
> be
> > an already defined constant (or keyword) in the message format of the
> > protocol.
> >
> > For example, if I have a protocol where “network-map”, “src”, and “dsts”
> > are defined keywords, JSON text A meets the condition whereas B does not
> > because “PID1” and “PID2” are not protocol keywords (they are resource
> IDs).
> >
> > A:
> >
> > {
> >
> > “network-map”: [
> >
> > {
> >
> > “src”: “PID1”,
> >
> > “dsts”: [“PID2”, “PID3”]
> >
> > },
> >
> > {
> >
> > “src”: “PID2”,
> >
> > “dsts”: [“PID3”, “PID4”]
> >
> > }
> >
> > ]
> >
> > }
> >
> > B:
> >
> >
> > {
> >
> > “network-map”: [
> >
> > “PID1: [“PID2”, “PID3”],
> >
> > “PID2”: [“PID3”, “PID4”]
> >
> > ]
> >
> > }
> >
> >>From this we know that since ALTO protocol uses encoding B, there cannot
> be
> > a syntactically equivalent YANG model.
> >
> > 2.2 The URI condition
> >
> > Some of the YANG-related protocols might have URI constraints, e.g.
> > RESTCONF. For now, we assume either that the JSON-based protocol URI
> could
> > be conformed to RESTCONF compliant uri, or that the server could have a
> > routing mapping between the protocol compliant uri and the RESTCONF
> > compliant uri, hence this condition would not be an issue, which allows
> us
> > to focus on the message encoding condition.
> >
> > 3. Proof
> >
> > 3.1. The message encoding condition is necessary.
> >
> > We first note that this condition is a necessary condition for a JSON
> based
> > protocol to have a syntactically equivalent YANG model by proving its
> > contrapositive.
> >
> > If one of the keys in the key-value pair in the JSON document is not
> > pre-defined, the corresponding XML tags will not be pre-defined keywords.
> > Therefore, it would not possible to model it in YANG without using
> > YANG’s anyxml
> > statement (which allows arbitrary XML content).
> >
> > However, using the anyxml statement would defeat our purpose of modeling
> > the data as it allows arbitrary XML content, and will not be helpful in
> the
> > subsequent parsing process.
> >
> > 3.2. The message encoding condition is sufficient.
> >
> > We prove this by providing a translation procedure from a JSON message
> that
> > is compliant with the protocol we are trying to model, to a custom java
> > class that can be used for Jackson data binding, then to a YANG model.
> >
> > We note that the middle step of translating to and from the custom parser
> > class is not necessary, but it will be useful.
> >
> > 3.2.1. motivation
> >
> > JSON data binding is the process of binding data structures (objects,
> > arrays, etc.) in JSON to the appropriate data structures in the server
> > (e.g. java classes, database tables, etc.) in parsing the JSON text. In
> > order to process JSON messages in a meaningful manner, data binding is
> > necessary. Even if the binding is not explicit, the server would need to
> do
> > it eventually. For example, one can read JSON content in a stream without
> > binding it to the java classes, but eventually in order to make sense of
> > the data, the server would eventually have to organize it, which is
> roughly
> > analogous to data binding upfront. Popular choices for JSON parsing and
> > data binding include jackson and gson.
> >
> > We use Jackson full data binding as our approach. Full data binding binds
> > JSON content into plain old java objects (POJOs), i.e. this custom parser
> > class can neither extend nor implement any other class. Jackson uses
> > ObjectMapper with the custom parser class to parse JSON content into this
> > class.
> >
> > 3.2.2. The message encoding condition
> >
> > The message encoding condition is that all keys in each key-value pair in
> > the JSON text must be pre-defined keywords. As the keys will become
> either
> > class names and instance variable names, or be keys in the java maps, it
> is
> > easy to see that the condition is equivalent to “there exists a full data
> > binding in Jackson custom parser class without using any map structures
> > (Map<String, ?>).”
> >
> > 3.2.3. Proof
> >
> > We provide a recursive binding process from a JSON object to the Jackson
> > custom parser class to be used by Jackson ObjectMapper.
> >
> > Type determine_type(value) {
> >
> > if (value is of a primitive type, i.e. string, number, boolean, null) {
> >
> > return the corresponding java primitive type;
> >
> > }
> >
> > if (value is a JSON object) {
> >
> > return build_parser_class(value).class;
> >
> > }
> >
> > if (value is an array) {
> >
> > return ArrayList<T> where T=determine_type(value[0]);
> >
> > }
> >
> > // should not reach here.
> >
> > }
> >
> > Class build_parser_class(JSONObject obj) {
> >
> > create custom class C;
> >
> > for each key/value pair in the obj {
> >
> > add instance variable v in C;
> >
> > the name of variable v <- key; // (__known__ a/c to our assumption)
> >
> > the type of variable v <- determine_type(value);
> >
> > }
> >
> > return C.class;
> >
> > }
> >
> > Naming:
> >
> > --change everything into CamelCase (i.e. remove dashes, etc.)
> >
> > --for instance variables, use “my” prefix, (e.g. myVariable,
> myNetworkMap,
> > etc.)
> >
> > --for the custom class name, if the object is an element of the array,
> use
> > “Element” suffix.
> >
> > This is just one convention so that the next step proceeds smoothly. As
> > long as this naming translation is consistent with the naming stage in
> the
> > next step, it will work just fine.
> >
> > Example (a modified ALTO protocol network map example):
> >
> > JSON object:
> >
> > {
> >
> > “meta”: {
> >
> > “resource-id”: “my-default-map”,
> >
> > “tag”: “aab875ef69c87d012”
> >
> > },
> >
> > “network-map”: [
> >
> > {
> >
> > “src”: “PID1”,
> >
> > “dsts”: [“PID1”, “PID2”, “PID3”]
> >
> > },
> >
> > {
> >
> > “src”: “PID2”,
> >
> > “dsts”: [“PID1”, “PID3”]
> >
> > },
> >
> > {
> >
> > “src”: “PID3”,
> >
> > “dsts”: [“PID2”, “PID3”]
> >
> > }
> >
> > ]
> >
> > }
> >
> > Result of build_parser_class(obj):
> >
> > Class JSONObject {
> >
> > Meta myMeta;
> >
> > ArrayList<NetworkMapElement> myNetworkMap;
> >
> > }
> >
> > Class Meta {
> >
> > String myResourceId;
> >
> > String myTag;
> >
> > }
> >
> > Class NetworkMapElement {
> >
> > String mySrc;
> >
> > ArrayList<String> myDsts;
> >
> > }
> >
> > Now given the Jackson Parser Java Class, to get a syntactically
> equivalent
> > YANG model:
> >
> > YANGModel build_yang_model(Class C) {
> >
> > for each instance variable (Type, Name) {
> >
> > if (Type is primitive type: string, number, boolean, null) {
> >
> > add the following to the YANG module:
> >
> > “leaf Name { type <YANG equivalent of Type>; }”
> >
> > }
> >
> > if (Type is an ArrayList<TypeElement>) {
> >
> > if (TypeElement is primitive type) {
> >
> > add the following to the YANG module:
> >
> > “leaf-list Name { type <YANG equivalent of TypeElement>; }”
> >
> > } else {
> >
> > // TypeElement is a custom parser class
> >
> > add the following to the YANG module:
> >
> > “list Name { <result from build_yang_model<TypeElement.class>> }”
> >
> > }
> >
> > }
> >
> > if (Type is a custom parser class) {
> >
> > add the following to the YANG module:
> >
> > “container Name { <result from build_yang_model<Type.class>> }”
> >
> > }
> >
> > }
> >
> > }
> >
> > Result from the previous example:
> >
> > container meta {
> >
> > leaf resource-id {
> >
> > type string;
> >
> > }
> >
> > leaf tag {
> >
> > type string;
> >
> > }
> >
> > }
> >
> > list network-map {
> >
> > leaf src {
> >
> > type string;
> >
> > }
> >
> > leaf-list dsts {
> >
> > type string;
> >
> > }
> >
> > }
> >
> > This does validate the JSON document with XML-JSON encoding. For your
> > reference, this is the XML document which validates:
> >
> > <?xml version="1.0" encoding="UTF-8" ?>
> >
> > <meta>
> >
> > <resource-id>my-default-map</resource-id>
> >
> > <tag>aab875ef69c87d012</tag>
> >
> > </meta>
> >
> > <network-map>
> >
> > <src>PID1</src>
> >
> > <dsts>PID1</dsts>
> >
> > <dsts>PID2</dsts>
> >
> > <dsts>PID3</dsts>
> >
> > </network-map>
> >
> > <network-map>
> >
> > <src>PID2</src>
> >
> > <dsts>PID1</dsts>
> >
> > <dsts>PID3</dsts>
> >
> > </network-map>
> >
> > <network-map>
> >
> > <src>PID3</src>
> >
> > <dsts>PID2</dsts>
> >
> > <dsts>PID3</dsts>
> >
> > </network-map>
> >
> > This proves that the message encoding condition is a sufficient condition
> > for the JSON object to have a YANG model.
> >
> > Note the model generated is very crude and lose almost all constraints
> and
> > all inheritance features (if any), because it focuses on the syntax and
> is
> > essentially converted from an JSON object compliant with a protocol
> instead
> > of from the protocol itself. Hence this result is more useful in
> > determining which JSON based protocols cannot have a syntactically
> > equivalent YANG model, than in generating a good YANG model.
> >
> > 3.3. Conclusion
> >
> > Our claim holds. A JSON based protocol carried by HTTP can have
> > syntactically equivalent YANG model, if and only if all the keys in the
> > key-value pairs in the JSON message are pre-defined keywords.
> >
> > 4. Semantic equivalence
> >
> > For JSON based protocols that don’t satisfy the message encoding
> condition,
> > it is still possible to have a semantically equivalent YANG model. All
> that
> > is required for the protocol compliant clients and the YANG model
> compliant
> > server to interoperate is an adapter which does the following:
> >
> > 1) translate FROM YANG server compliant response msg TO alto compliant
> > response msg
> >
> > 2) translate FROM alto compliant request msg TO YANG server compliant
> > request msg
> >
> > 4.1. Claim
> >
> > This adapter needs to be protocol-aware.
> >
> > Ideally, given any YANG model, we would like to be able to automatically
> > (or at least mechanically) generate this message adapter, which means not
> > looking at the protocol or its compliant msgs. However, without knowing
> the
> > specific protocol that we are working with (i.e. human intervention, i.e.
> > looking at the protocol compliant msgs), such an adapter cannot be
> > auto-generated.
> >
> > 4.2. Proof by Indistinguishability
> >
> > Suppose both the YANG server compliant msg m_y and the actually protocol
> > compliant msg m_p are in JSON (or have been encoded into JSON). Looking
> at
> > the differences between the two messages, call these differences {d1, d2,
> > ..., dn}. The goal for the auto-generated adapter would be to identify
> and
> > eliminate these differences. Construct a new JSON msg m' where all but
> one
> > difference di is the same as m_p and di is the same as the m_y. Without
> > looking at the protocol (or m_p), the auto-generated adapter would not be
> > able to distinguish between m' and m_p in its translation process, which
> > means, it won't be able to tell whether it should change di or not.
> Hence,
> > such an adapter must be protocol-aware.
> >
> > A good example is the dependent-vtag in the ALTO protocol:
> >
> > "dependent-vtag" : [
> >
> > {
> >
> > "resource-id" : "my-network-map",
> >
> > "tag" : "abcd1234"
> >
> > }
> >
> > ]
> >
> > It was specified this way in the alto protocol. However, it could
> > conceivably be the case that it was originally the following map
> structure,
> > and was converted into the above encoding because of the map->list+key
> > issue. (This case is actually one of the few differences in the m_y and
> m_p
> > where the adapter does not need to convert it back to a map structure.)
> >
> > "dependent-vtag" : {
> >
> > "my-network-map" : {
> >
> > "tag" : "abcd1234"
> >
> > }
> >
> > }
> >
> > Without knowing the protocol, there is no way to tell.
> >
> > 5. Ramifications
> >
> > We now understand the basic condition for a JSON based protocol to have a
> > YANG Model. For the protocols that don’t meet this condition, there can
> be
> > a semantic equivalent YANG model, but there won’t be a generic process of
> > generating the adapter for all such protocols.
> > _______________________________________________
> > netmod mailing list
> > [email protected]
> > https://www.ietf.org/mailman/listinfo/netmod
>
> --
> Ladislav Lhotka, CZ.NIC Labs
> PGP Key ID: E74E8C0C
>
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
