Hi Xiao, see inline.
Xiao SHI <[email protected]> writes: > Hi Lada, > > I agree with your statement that YANG is designed to model datastore and > the protocol messages. > > Right now the question I am trying to answer is this scenario: we have the > protocols (hence the protocol messages) first, and would like to know > whether we would be able to use YANG to model the content messages (in > order to use the YANG tools for this protocol, etc.). And the question is > what are the conditions for JSON msgs to be able to be modeled by YANG. > > For JSON arrays: > 0. a standalone non-object JSON value is not model-able in YANG: > examples: > --"foo" > --14 > --[1,2,3] > which translates to the following in XML: > <0>1</0> > <1>2</1> > <2>3</2> > > This is why your example "foo": [ { "bar": 1 }, 2 ] cannot be > modeled--because 2 is a standalone value and {"bar": 1} is an object, Yes. > otherwise there are following ways to model an array with a mix of elements: > > 1. an array with only built-in types or derived types (standalone JSON > values except for arrays and objects) can be modeled in YANG as the > following: > "foo" : [1, "abc", enum1] It should be "foo" : [1, "abc", "enum1"] > > leaf-list foo { > type union { > type int64; > type string; > type enumeration { > enum enum1; > } > } > } Right. > > 2. an array of all objects > (1) This structure is model-able as well, right? > "foo": [{"a": 1}, {"a": 1, "b": 2}] > > list foo { > leaf a { > type int; > } > leaf b { > type int; > } > } Yes. > > (2) how about this? > "foo": [{"a": 1, "b": 2, "c": 3}, {"c": 1, "b":3, "y": 2}] > > If the JSON-XML translate it to the following: > <foo> > <a>1</a> > <b>2</b> > <c>3</c> > </foo> > <foo> > <c>1</c> > <b>3</b> > <y>2</y> > </foo> > We can still model this using the choice statement, right? Right. > > If the above statements about whether these JSON texts are model-able are > correct, essentially the only JSON arrays that are not model-able are (a) > those mixed with objects and a standalone non-object JSON value; and (b) > those whose elements is an array. And this gap is caused by the difference > between leaf-list modeling capabilities of handling arrays with non-object, > non-array, standalone JSON values, and the list+choice modeling > capabilities of modeling arrays with objects. Does this sound > plausible? It does, I am not aware of any other issues. Cheers, Lada > > Cheers, > Xiao > > On Mon, Oct 20, 2014 at 11:56 AM, Ladislav Lhotka <[email protected]> wrote: > >> Xiao SHI <[email protected]> writes: >> >> > Hi Lada, >> > >> > Thank you so much for your feedback! >> > >> > The motivation here is to produce a YANG model given a defined protocol, >> > e.g. ALTO, CDNi, etc. This way, we may incorporate future protocols into >> > ODL controllers or use other YANG related tools to handle those >> protocols. >> > Hence the objective of the modeling is the protocol, not the messages. I >> am >> > not suggesting to use YANG as a schema language. >> >> Hmm, YANG was designed for modelling datastores and contents of protocol >> messages, not protocols as such. >> >> > >> > The conditions discussed in this process will be helpful in the following >> > sense: >> > 1) disprove the possibility to get a syntactically equivalent YANG model >> > from a JSON based protocol; >> > 2) generate a Jackson JSON parser class and YANG model from the protocols >> > that meet the criteria; >> > 3) for designers of future protocols who are thinking of using YANG >> related >> > tools, set a guideline of what their messages should look like (or MUST >> not >> > contain); >> > 4) for the YANG/NETCONF WG, describing the compatibility issues that they >> > might be interested in tackling for expanding YANG's usage. >> > >> > For this example, "foo": [ { "bar": 1 }, 2 ] is also not easy to do >> > data-binding in the parser, because the elements of the array are not of >> > the same type, which is indeed an assumption that I left out. >> > >> > For the {"foo”:null} example, can't we model it using type empty? That's >> > equivalent to the XML element <foo /> right? >> >> <foo/> corresponds to >> >> "foo": [null] >> >> Section 6.9 in draft-ietf-netmod-yang-json-01 gives the reasoning, >> essentially it is because some software isn't able to distinguish >> >> "foo": null >> >> from "foo" being absent. >> >> Lada >> >> > >> > Cheers, >> > Xiao >> > >> > On Mon, Oct 20, 2014 at 9:23 AM, Ladislav Lhotka <[email protected]> wrote: >> > >> >> Xiao SHI <[email protected]> writes: >> >> >> >> > Hi folks, >> >> > >> >> > >> >> > As a few of us were working on modeling the ALTO protocol using YANG, >> we >> >> > were pondering on a more general question: can YANG model all JSON >> based >> >> > protocols? What is the condition for a JSON based protocol (at least >> the >> >> > message format) to have an syntactically equivalent, hence >> interoperable, >> >> > YANG model with JSON encoding? Alternatively, in order to >> interoperate, >> >> > semantic equivalence might be sufficient, is there any condition for >> >> > semantic equivalence? >> >> > >> >> > >> >> > tl;dr: A JSON based protocol carried by HTTP can have syntactically >> >> > equivalent YANG model, if and only if all the keys in the key-value >> pairs >> >> > in the JSON message are pre-defined keywords in the protocol. >> >> > >> >> >> >> Generating a YANG data model from existing instance data seems backwards >> >> to me, although Examplotron (http://examplotron.org) does something >> quite >> >> similar. In any case, it might be quite challenging because YANG isn't >> >> intended as a general schema language. Your observation above is true >> >> but it is just one aspect of the problem. For instance >> >> >> >> "foo": [ { "bar": 1 }, 2 ] >> >> >> >> or >> >> >> >> "foo": null >> >> >> >> cannot be modelled in YANG using the encoding of >> >> draft-ietf-netmod-yang-json. >> >> >> >> Lada >> >> >> >> > >> >> > We've come up with a few ideas, and a few things below are work in >> >> > progress, but we would love your feedback! >> >> > >> >> > >> >> > Thank you! >> >> > >> >> > Xiao >> >> > >> >> > >> >> > ====================== >> >> > >> >> > 1. Introduction. >> >> > >> >> > JavaScript Object Notation (JSON) has been a popular choice as the >> >> message >> >> > encoding for many network protocols such as the Application Layer >> Traffic >> >> > Optimization (ALTO) protocol, the Content Delivery Networks >> >> Interconnection >> >> > (CDNi) protocol, etc. >> >> > >> >> > Meanwhile, there are broad interests in the networking community to >> use >> >> > YANG to define data model so that one can use YANG related tools such >> as >> >> > OpenDayLight controller. Although YANG itself is XML based, there have >> >> been >> >> > efforts to model JSON content using YANG >> >> [draft-ietf-netmod-yang-json-01]. >> >> > >> >> > A natural question rises: can YANG model all JSON based protocols? >> What >> >> is >> >> > the condition for a JSON based protocol (at least the message format) >> to >> >> > have an syntactically equivalent, hence interoperable, YANG model with >> >> JSON >> >> > encoding? Alternatively, in order to interoperate, semantic >> equivalence >> >> > might be sufficient, is there any condition for semantic equivalence? >> >> > >> >> > 2. Claim >> >> > >> >> > A JSON based protocol carried by HTTP can have syntactically >> equivalent >> >> > YANG model, if and only if: >> >> > >> >> > (1) the message encoding condition is met; >> >> > >> >> > (2) the uri condition is met. >> >> > >> >> > 2.1. The message encoding condition >> >> > >> >> > The JSON message encoding MUST not contain a variable as a key in a >> JSON >> >> > object key-value pair. In other words, the keys in the JSON message >> must >> >> be >> >> > an already defined constant (or keyword) in the message format of the >> >> > protocol. >> >> > >> >> > For example, if I have a protocol where “network-map”, “src”, and >> “dsts” >> >> > are defined keywords, JSON text A meets the condition whereas B does >> not >> >> > because “PID1” and “PID2” are not protocol keywords (they are resource >> >> IDs). >> >> > >> >> > A: >> >> > >> >> > { >> >> > >> >> > “network-map”: [ >> >> > >> >> > { >> >> > >> >> > “src”: “PID1”, >> >> > >> >> > “dsts”: [“PID2”, “PID3”] >> >> > >> >> > }, >> >> > >> >> > { >> >> > >> >> > “src”: “PID2”, >> >> > >> >> > “dsts”: [“PID3”, “PID4”] >> >> > >> >> > } >> >> > >> >> > ] >> >> > >> >> > } >> >> > >> >> > B: >> >> > >> >> > >> >> > { >> >> > >> >> > “network-map”: [ >> >> > >> >> > “PID1: [“PID2”, “PID3”], >> >> > >> >> > “PID2”: [“PID3”, “PID4”] >> >> > >> >> > ] >> >> > >> >> > } >> >> > >> >> >>From this we know that since ALTO protocol uses encoding B, there >> cannot >> >> be >> >> > a syntactically equivalent YANG model. >> >> > >> >> > 2.2 The URI condition >> >> > >> >> > Some of the YANG-related protocols might have URI constraints, e.g. >> >> > RESTCONF. For now, we assume either that the JSON-based protocol URI >> >> could >> >> > be conformed to RESTCONF compliant uri, or that the server could have >> a >> >> > routing mapping between the protocol compliant uri and the RESTCONF >> >> > compliant uri, hence this condition would not be an issue, which >> allows >> >> us >> >> > to focus on the message encoding condition. >> >> > >> >> > 3. Proof >> >> > >> >> > 3.1. The message encoding condition is necessary. >> >> > >> >> > We first note that this condition is a necessary condition for a JSON >> >> based >> >> > protocol to have a syntactically equivalent YANG model by proving its >> >> > contrapositive. >> >> > >> >> > If one of the keys in the key-value pair in the JSON document is not >> >> > pre-defined, the corresponding XML tags will not be pre-defined >> keywords. >> >> > Therefore, it would not possible to model it in YANG without using >> >> > YANG’s anyxml >> >> > statement (which allows arbitrary XML content). >> >> > >> >> > However, using the anyxml statement would defeat our purpose of >> modeling >> >> > the data as it allows arbitrary XML content, and will not be helpful >> in >> >> the >> >> > subsequent parsing process. >> >> > >> >> > 3.2. The message encoding condition is sufficient. >> >> > >> >> > We prove this by providing a translation procedure from a JSON message >> >> that >> >> > is compliant with the protocol we are trying to model, to a custom >> java >> >> > class that can be used for Jackson data binding, then to a YANG model. >> >> > >> >> > We note that the middle step of translating to and from the custom >> parser >> >> > class is not necessary, but it will be useful. >> >> > >> >> > 3.2.1. motivation >> >> > >> >> > JSON data binding is the process of binding data structures (objects, >> >> > arrays, etc.) in JSON to the appropriate data structures in the server >> >> > (e.g. java classes, database tables, etc.) in parsing the JSON text. >> In >> >> > order to process JSON messages in a meaningful manner, data binding is >> >> > necessary. Even if the binding is not explicit, the server would need >> to >> >> do >> >> > it eventually. For example, one can read JSON content in a stream >> without >> >> > binding it to the java classes, but eventually in order to make sense >> of >> >> > the data, the server would eventually have to organize it, which is >> >> roughly >> >> > analogous to data binding upfront. Popular choices for JSON parsing >> and >> >> > data binding include jackson and gson. >> >> > >> >> > We use Jackson full data binding as our approach. Full data binding >> binds >> >> > JSON content into plain old java objects (POJOs), i.e. this custom >> parser >> >> > class can neither extend nor implement any other class. Jackson uses >> >> > ObjectMapper with the custom parser class to parse JSON content into >> this >> >> > class. >> >> > >> >> > 3.2.2. The message encoding condition >> >> > >> >> > The message encoding condition is that all keys in each key-value >> pair in >> >> > the JSON text must be pre-defined keywords. As the keys will become >> >> either >> >> > class names and instance variable names, or be keys in the java maps, >> it >> >> is >> >> > easy to see that the condition is equivalent to “there exists a full >> data >> >> > binding in Jackson custom parser class without using any map >> structures >> >> > (Map<String, ?>).” >> >> > >> >> > 3.2.3. Proof >> >> > >> >> > We provide a recursive binding process from a JSON object to the >> Jackson >> >> > custom parser class to be used by Jackson ObjectMapper. >> >> > >> >> > Type determine_type(value) { >> >> > >> >> > if (value is of a primitive type, i.e. string, number, boolean, >> null) { >> >> > >> >> > return the corresponding java primitive type; >> >> > >> >> > } >> >> > >> >> > if (value is a JSON object) { >> >> > >> >> > return build_parser_class(value).class; >> >> > >> >> > } >> >> > >> >> > if (value is an array) { >> >> > >> >> > return ArrayList<T> where T=determine_type(value[0]); >> >> > >> >> > } >> >> > >> >> > // should not reach here. >> >> > >> >> > } >> >> > >> >> > Class build_parser_class(JSONObject obj) { >> >> > >> >> > create custom class C; >> >> > >> >> > for each key/value pair in the obj { >> >> > >> >> > add instance variable v in C; >> >> > >> >> > the name of variable v <- key; // (__known__ a/c to our assumption) >> >> > >> >> > the type of variable v <- determine_type(value); >> >> > >> >> > } >> >> > >> >> > return C.class; >> >> > >> >> > } >> >> > >> >> > Naming: >> >> > >> >> > --change everything into CamelCase (i.e. remove dashes, etc.) >> >> > >> >> > --for instance variables, use “my” prefix, (e.g. myVariable, >> >> myNetworkMap, >> >> > etc.) >> >> > >> >> > --for the custom class name, if the object is an element of the array, >> >> use >> >> > “Element” suffix. >> >> > >> >> > This is just one convention so that the next step proceeds smoothly. >> As >> >> > long as this naming translation is consistent with the naming stage in >> >> the >> >> > next step, it will work just fine. >> >> > >> >> > Example (a modified ALTO protocol network map example): >> >> > >> >> > JSON object: >> >> > >> >> > { >> >> > >> >> > “meta”: { >> >> > >> >> > “resource-id”: “my-default-map”, >> >> > >> >> > “tag”: “aab875ef69c87d012” >> >> > >> >> > }, >> >> > >> >> > “network-map”: [ >> >> > >> >> > { >> >> > >> >> > “src”: “PID1”, >> >> > >> >> > “dsts”: [“PID1”, “PID2”, “PID3”] >> >> > >> >> > }, >> >> > >> >> > { >> >> > >> >> > “src”: “PID2”, >> >> > >> >> > “dsts”: [“PID1”, “PID3”] >> >> > >> >> > }, >> >> > >> >> > { >> >> > >> >> > “src”: “PID3”, >> >> > >> >> > “dsts”: [“PID2”, “PID3”] >> >> > >> >> > } >> >> > >> >> > ] >> >> > >> >> > } >> >> > >> >> > Result of build_parser_class(obj): >> >> > >> >> > Class JSONObject { >> >> > >> >> > Meta myMeta; >> >> > >> >> > ArrayList<NetworkMapElement> myNetworkMap; >> >> > >> >> > } >> >> > >> >> > Class Meta { >> >> > >> >> > String myResourceId; >> >> > >> >> > String myTag; >> >> > >> >> > } >> >> > >> >> > Class NetworkMapElement { >> >> > >> >> > String mySrc; >> >> > >> >> > ArrayList<String> myDsts; >> >> > >> >> > } >> >> > >> >> > Now given the Jackson Parser Java Class, to get a syntactically >> >> equivalent >> >> > YANG model: >> >> > >> >> > YANGModel build_yang_model(Class C) { >> >> > >> >> > for each instance variable (Type, Name) { >> >> > >> >> > if (Type is primitive type: string, number, boolean, null) { >> >> > >> >> > add the following to the YANG module: >> >> > >> >> > “leaf Name { type <YANG equivalent of Type>; }” >> >> > >> >> > } >> >> > >> >> > if (Type is an ArrayList<TypeElement>) { >> >> > >> >> > if (TypeElement is primitive type) { >> >> > >> >> > add the following to the YANG module: >> >> > >> >> > “leaf-list Name { type <YANG equivalent of TypeElement>; }” >> >> > >> >> > } else { >> >> > >> >> > // TypeElement is a custom parser class >> >> > >> >> > add the following to the YANG module: >> >> > >> >> > “list Name { <result from build_yang_model<TypeElement.class>> >> }” >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > if (Type is a custom parser class) { >> >> > >> >> > add the following to the YANG module: >> >> > >> >> > “container Name { <result from build_yang_model<Type.class>> }” >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > Result from the previous example: >> >> > >> >> > container meta { >> >> > >> >> > leaf resource-id { >> >> > >> >> > type string; >> >> > >> >> > } >> >> > >> >> > leaf tag { >> >> > >> >> > type string; >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > list network-map { >> >> > >> >> > leaf src { >> >> > >> >> > type string; >> >> > >> >> > } >> >> > >> >> > leaf-list dsts { >> >> > >> >> > type string; >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > This does validate the JSON document with XML-JSON encoding. For your >> >> > reference, this is the XML document which validates: >> >> > >> >> > <?xml version="1.0" encoding="UTF-8" ?> >> >> > >> >> > <meta> >> >> > >> >> > <resource-id>my-default-map</resource-id> >> >> > >> >> > <tag>aab875ef69c87d012</tag> >> >> > >> >> > </meta> >> >> > >> >> > <network-map> >> >> > >> >> > <src>PID1</src> >> >> > >> >> > <dsts>PID1</dsts> >> >> > >> >> > <dsts>PID2</dsts> >> >> > >> >> > <dsts>PID3</dsts> >> >> > >> >> > </network-map> >> >> > >> >> > <network-map> >> >> > >> >> > <src>PID2</src> >> >> > >> >> > <dsts>PID1</dsts> >> >> > >> >> > <dsts>PID3</dsts> >> >> > >> >> > </network-map> >> >> > >> >> > <network-map> >> >> > >> >> > <src>PID3</src> >> >> > >> >> > <dsts>PID2</dsts> >> >> > >> >> > <dsts>PID3</dsts> >> >> > >> >> > </network-map> >> >> > >> >> > This proves that the message encoding condition is a sufficient >> condition >> >> > for the JSON object to have a YANG model. >> >> > >> >> > Note the model generated is very crude and lose almost all constraints >> >> and >> >> > all inheritance features (if any), because it focuses on the syntax >> and >> >> is >> >> > essentially converted from an JSON object compliant with a protocol >> >> instead >> >> > of from the protocol itself. Hence this result is more useful in >> >> > determining which JSON based protocols cannot have a syntactically >> >> > equivalent YANG model, than in generating a good YANG model. >> >> > >> >> > 3.3. Conclusion >> >> > >> >> > Our claim holds. A JSON based protocol carried by HTTP can have >> >> > syntactically equivalent YANG model, if and only if all the keys in >> the >> >> > key-value pairs in the JSON message are pre-defined keywords. >> >> > >> >> > 4. Semantic equivalence >> >> > >> >> > For JSON based protocols that don’t satisfy the message encoding >> >> condition, >> >> > it is still possible to have a semantically equivalent YANG model. All >> >> that >> >> > is required for the protocol compliant clients and the YANG model >> >> compliant >> >> > server to interoperate is an adapter which does the following: >> >> > >> >> > 1) translate FROM YANG server compliant response msg TO alto compliant >> >> > response msg >> >> > >> >> > 2) translate FROM alto compliant request msg TO YANG server compliant >> >> > request msg >> >> > >> >> > 4.1. Claim >> >> > >> >> > This adapter needs to be protocol-aware. >> >> > >> >> > Ideally, given any YANG model, we would like to be able to >> automatically >> >> > (or at least mechanically) generate this message adapter, which means >> not >> >> > looking at the protocol or its compliant msgs. However, without >> knowing >> >> the >> >> > specific protocol that we are working with (i.e. human intervention, >> i.e. >> >> > looking at the protocol compliant msgs), such an adapter cannot be >> >> > auto-generated. >> >> > >> >> > 4.2. Proof by Indistinguishability >> >> > >> >> > Suppose both the YANG server compliant msg m_y and the actually >> protocol >> >> > compliant msg m_p are in JSON (or have been encoded into JSON). >> Looking >> >> at >> >> > the differences between the two messages, call these differences {d1, >> d2, >> >> > ..., dn}. The goal for the auto-generated adapter would be to identify >> >> and >> >> > eliminate these differences. Construct a new JSON msg m' where all but >> >> one >> >> > difference di is the same as m_p and di is the same as the m_y. >> Without >> >> > looking at the protocol (or m_p), the auto-generated adapter would >> not be >> >> > able to distinguish between m' and m_p in its translation process, >> which >> >> > means, it won't be able to tell whether it should change di or not. >> >> Hence, >> >> > such an adapter must be protocol-aware. >> >> > >> >> > A good example is the dependent-vtag in the ALTO protocol: >> >> > >> >> > "dependent-vtag" : [ >> >> > >> >> > { >> >> > >> >> > "resource-id" : "my-network-map", >> >> > >> >> > "tag" : "abcd1234" >> >> > >> >> > } >> >> > >> >> > ] >> >> > >> >> > It was specified this way in the alto protocol. However, it could >> >> > conceivably be the case that it was originally the following map >> >> structure, >> >> > and was converted into the above encoding because of the map->list+key >> >> > issue. (This case is actually one of the few differences in the m_y >> and >> >> m_p >> >> > where the adapter does not need to convert it back to a map >> structure.) >> >> > >> >> > "dependent-vtag" : { >> >> > >> >> > "my-network-map" : { >> >> > >> >> > "tag" : "abcd1234" >> >> > >> >> > } >> >> > >> >> > } >> >> > >> >> > Without knowing the protocol, there is no way to tell. >> >> > >> >> > 5. Ramifications >> >> > >> >> > We now understand the basic condition for a JSON based protocol to >> have a >> >> > YANG Model. For the protocols that don’t meet this condition, there >> can >> >> be >> >> > a semantic equivalent YANG model, but there won’t be a generic >> process of >> >> > generating the adapter for all such protocols. >> >> > _______________________________________________ >> >> > netmod mailing list >> >> > [email protected] >> >> > https://www.ietf.org/mailman/listinfo/netmod >> >> >> >> -- >> >> Ladislav Lhotka, CZ.NIC Labs >> >> PGP Key ID: E74E8C0C >> >> >> >> -- >> Ladislav Lhotka, CZ.NIC Labs >> PGP Key ID: E74E8C0C >> -- Ladislav Lhotka, CZ.NIC Labs PGP Key ID: E74E8C0C _______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
