Hi Andy, I agree with you. Just to clarify, what I am trying to do is to use YANG data structures for the protocol whose message encoding is JSON. I am not trying to use JSON as a data structure. Does this make sense?
Best, Xiao On Mon, Oct 20, 2014 at 11:29 AM, Andy Bierman <[email protected]> wrote: > > > On Mon, Oct 20, 2014 at 6:23 AM, Ladislav Lhotka <[email protected]> wrote: > >> Xiao SHI <[email protected]> writes: >> >> > Hi folks, >> > >> > >> > As a few of us were working on modeling the ALTO protocol using YANG, we >> > were pondering on a more general question: can YANG model all JSON based >> > protocols? What is the condition for a JSON based protocol (at least the >> > message format) to have an syntactically equivalent, hence >> interoperable, >> > YANG model with JSON encoding? Alternatively, in order to interoperate, >> > semantic equivalence might be sufficient, is there any condition for >> > semantic equivalence? >> > >> > >> > tl;dr: A JSON based protocol carried by HTTP can have syntactically >> > equivalent YANG model, if and only if all the keys in the key-value >> pairs >> > in the JSON message are pre-defined keywords in the protocol. >> > >> >> Generating a YANG data model from existing instance data seems backwards >> to me, although Examplotron (http://examplotron.org) does something quite >> similar. In any case, it might be quite challenging because YANG isn't >> intended as a general schema language. Your observation above is true >> but it is just one aspect of the problem. For instance >> >> "foo": [ { "bar": 1 }, 2 ] >> >> or >> >> "foo": null >> >> cannot be modelled in YANG using the encoding of >> draft-ietf-netmod-yang-json. >> >> > > I think perhaps RESTCONF is overloading JSON and this has led to some > confusion. > RESTCONF uses JSON strictly as an encoding format, not a data structure. > JSON is just > the message encoding. The corresponding instance document must conform to > the YANG definitions (not just be valid JSON). > > A protocol should pick between YANG data structures and JSON data > structures, > and not try to do both. > > > > >> Lada >> >> > Andy > > >> > >> > We've come up with a few ideas, and a few things below are work in >> > progress, but we would love your feedback! >> > >> > >> > Thank you! >> > >> > Xiao >> > >> > >> > ====================== >> > >> > 1. Introduction. >> > >> > JavaScript Object Notation (JSON) has been a popular choice as the >> message >> > encoding for many network protocols such as the Application Layer >> Traffic >> > Optimization (ALTO) protocol, the Content Delivery Networks >> Interconnection >> > (CDNi) protocol, etc. >> > >> > Meanwhile, there are broad interests in the networking community to use >> > YANG to define data model so that one can use YANG related tools such as >> > OpenDayLight controller. Although YANG itself is XML based, there have >> been >> > efforts to model JSON content using YANG >> [draft-ietf-netmod-yang-json-01]. >> > >> > A natural question rises: can YANG model all JSON based protocols? What >> is >> > the condition for a JSON based protocol (at least the message format) to >> > have an syntactically equivalent, hence interoperable, YANG model with >> JSON >> > encoding? Alternatively, in order to interoperate, semantic equivalence >> > might be sufficient, is there any condition for semantic equivalence? >> > >> > 2. Claim >> > >> > A JSON based protocol carried by HTTP can have syntactically equivalent >> > YANG model, if and only if: >> > >> > (1) the message encoding condition is met; >> > >> > (2) the uri condition is met. >> > >> > 2.1. The message encoding condition >> > >> > The JSON message encoding MUST not contain a variable as a key in a JSON >> > object key-value pair. In other words, the keys in the JSON message >> must be >> > an already defined constant (or keyword) in the message format of the >> > protocol. >> > >> > For example, if I have a protocol where “network-map”, “src”, and “dsts” >> > are defined keywords, JSON text A meets the condition whereas B does not >> > because “PID1” and “PID2” are not protocol keywords (they are resource >> IDs). >> > >> > A: >> > >> > { >> > >> > “network-map”: [ >> > >> > { >> > >> > “src”: “PID1”, >> > >> > “dsts”: [“PID2”, “PID3”] >> > >> > }, >> > >> > { >> > >> > “src”: “PID2”, >> > >> > “dsts”: [“PID3”, “PID4”] >> > >> > } >> > >> > ] >> > >> > } >> > >> > B: >> > >> > >> > { >> > >> > “network-map”: [ >> > >> > “PID1: [“PID2”, “PID3”], >> > >> > “PID2”: [“PID3”, “PID4”] >> > >> > ] >> > >> > } >> > >> >>From this we know that since ALTO protocol uses encoding B, there >> cannot be >> > a syntactically equivalent YANG model. >> > >> > 2.2 The URI condition >> > >> > Some of the YANG-related protocols might have URI constraints, e.g. >> > RESTCONF. For now, we assume either that the JSON-based protocol URI >> could >> > be conformed to RESTCONF compliant uri, or that the server could have a >> > routing mapping between the protocol compliant uri and the RESTCONF >> > compliant uri, hence this condition would not be an issue, which allows >> us >> > to focus on the message encoding condition. >> > >> > 3. Proof >> > >> > 3.1. The message encoding condition is necessary. >> > >> > We first note that this condition is a necessary condition for a JSON >> based >> > protocol to have a syntactically equivalent YANG model by proving its >> > contrapositive. >> > >> > If one of the keys in the key-value pair in the JSON document is not >> > pre-defined, the corresponding XML tags will not be pre-defined >> keywords. >> > Therefore, it would not possible to model it in YANG without using >> > YANG’s anyxml >> > statement (which allows arbitrary XML content). >> > >> > However, using the anyxml statement would defeat our purpose of modeling >> > the data as it allows arbitrary XML content, and will not be helpful in >> the >> > subsequent parsing process. >> > >> > 3.2. The message encoding condition is sufficient. >> > >> > We prove this by providing a translation procedure from a JSON message >> that >> > is compliant with the protocol we are trying to model, to a custom java >> > class that can be used for Jackson data binding, then to a YANG model. >> > >> > We note that the middle step of translating to and from the custom >> parser >> > class is not necessary, but it will be useful. >> > >> > 3.2.1. motivation >> > >> > JSON data binding is the process of binding data structures (objects, >> > arrays, etc.) in JSON to the appropriate data structures in the server >> > (e.g. java classes, database tables, etc.) in parsing the JSON text. In >> > order to process JSON messages in a meaningful manner, data binding is >> > necessary. Even if the binding is not explicit, the server would need >> to do >> > it eventually. For example, one can read JSON content in a stream >> without >> > binding it to the java classes, but eventually in order to make sense of >> > the data, the server would eventually have to organize it, which is >> roughly >> > analogous to data binding upfront. Popular choices for JSON parsing and >> > data binding include jackson and gson. >> > >> > We use Jackson full data binding as our approach. Full data binding >> binds >> > JSON content into plain old java objects (POJOs), i.e. this custom >> parser >> > class can neither extend nor implement any other class. Jackson uses >> > ObjectMapper with the custom parser class to parse JSON content into >> this >> > class. >> > >> > 3.2.2. The message encoding condition >> > >> > The message encoding condition is that all keys in each key-value pair >> in >> > the JSON text must be pre-defined keywords. As the keys will become >> either >> > class names and instance variable names, or be keys in the java maps, >> it is >> > easy to see that the condition is equivalent to “there exists a full >> data >> > binding in Jackson custom parser class without using any map structures >> > (Map<String, ?>).” >> > >> > 3.2.3. Proof >> > >> > We provide a recursive binding process from a JSON object to the Jackson >> > custom parser class to be used by Jackson ObjectMapper. >> > >> > Type determine_type(value) { >> > >> > if (value is of a primitive type, i.e. string, number, boolean, null) { >> > >> > return the corresponding java primitive type; >> > >> > } >> > >> > if (value is a JSON object) { >> > >> > return build_parser_class(value).class; >> > >> > } >> > >> > if (value is an array) { >> > >> > return ArrayList<T> where T=determine_type(value[0]); >> > >> > } >> > >> > // should not reach here. >> > >> > } >> > >> > Class build_parser_class(JSONObject obj) { >> > >> > create custom class C; >> > >> > for each key/value pair in the obj { >> > >> > add instance variable v in C; >> > >> > the name of variable v <- key; // (__known__ a/c to our assumption) >> > >> > the type of variable v <- determine_type(value); >> > >> > } >> > >> > return C.class; >> > >> > } >> > >> > Naming: >> > >> > --change everything into CamelCase (i.e. remove dashes, etc.) >> > >> > --for instance variables, use “my” prefix, (e.g. myVariable, >> myNetworkMap, >> > etc.) >> > >> > --for the custom class name, if the object is an element of the array, >> use >> > “Element” suffix. >> > >> > This is just one convention so that the next step proceeds smoothly. As >> > long as this naming translation is consistent with the naming stage in >> the >> > next step, it will work just fine. >> > >> > Example (a modified ALTO protocol network map example): >> > >> > JSON object: >> > >> > { >> > >> > “meta”: { >> > >> > “resource-id”: “my-default-map”, >> > >> > “tag”: “aab875ef69c87d012” >> > >> > }, >> > >> > “network-map”: [ >> > >> > { >> > >> > “src”: “PID1”, >> > >> > “dsts”: [“PID1”, “PID2”, “PID3”] >> > >> > }, >> > >> > { >> > >> > “src”: “PID2”, >> > >> > “dsts”: [“PID1”, “PID3”] >> > >> > }, >> > >> > { >> > >> > “src”: “PID3”, >> > >> > “dsts”: [“PID2”, “PID3”] >> > >> > } >> > >> > ] >> > >> > } >> > >> > Result of build_parser_class(obj): >> > >> > Class JSONObject { >> > >> > Meta myMeta; >> > >> > ArrayList<NetworkMapElement> myNetworkMap; >> > >> > } >> > >> > Class Meta { >> > >> > String myResourceId; >> > >> > String myTag; >> > >> > } >> > >> > Class NetworkMapElement { >> > >> > String mySrc; >> > >> > ArrayList<String> myDsts; >> > >> > } >> > >> > Now given the Jackson Parser Java Class, to get a syntactically >> equivalent >> > YANG model: >> > >> > YANGModel build_yang_model(Class C) { >> > >> > for each instance variable (Type, Name) { >> > >> > if (Type is primitive type: string, number, boolean, null) { >> > >> > add the following to the YANG module: >> > >> > “leaf Name { type <YANG equivalent of Type>; }” >> > >> > } >> > >> > if (Type is an ArrayList<TypeElement>) { >> > >> > if (TypeElement is primitive type) { >> > >> > add the following to the YANG module: >> > >> > “leaf-list Name { type <YANG equivalent of TypeElement>; }” >> > >> > } else { >> > >> > // TypeElement is a custom parser class >> > >> > add the following to the YANG module: >> > >> > “list Name { <result from build_yang_model<TypeElement.class>> }” >> > >> > } >> > >> > } >> > >> > if (Type is a custom parser class) { >> > >> > add the following to the YANG module: >> > >> > “container Name { <result from build_yang_model<Type.class>> }” >> > >> > } >> > >> > } >> > >> > } >> > >> > Result from the previous example: >> > >> > container meta { >> > >> > leaf resource-id { >> > >> > type string; >> > >> > } >> > >> > leaf tag { >> > >> > type string; >> > >> > } >> > >> > } >> > >> > list network-map { >> > >> > leaf src { >> > >> > type string; >> > >> > } >> > >> > leaf-list dsts { >> > >> > type string; >> > >> > } >> > >> > } >> > >> > This does validate the JSON document with XML-JSON encoding. For your >> > reference, this is the XML document which validates: >> > >> > <?xml version="1.0" encoding="UTF-8" ?> >> > >> > <meta> >> > >> > <resource-id>my-default-map</resource-id> >> > >> > <tag>aab875ef69c87d012</tag> >> > >> > </meta> >> > >> > <network-map> >> > >> > <src>PID1</src> >> > >> > <dsts>PID1</dsts> >> > >> > <dsts>PID2</dsts> >> > >> > <dsts>PID3</dsts> >> > >> > </network-map> >> > >> > <network-map> >> > >> > <src>PID2</src> >> > >> > <dsts>PID1</dsts> >> > >> > <dsts>PID3</dsts> >> > >> > </network-map> >> > >> > <network-map> >> > >> > <src>PID3</src> >> > >> > <dsts>PID2</dsts> >> > >> > <dsts>PID3</dsts> >> > >> > </network-map> >> > >> > This proves that the message encoding condition is a sufficient >> condition >> > for the JSON object to have a YANG model. >> > >> > Note the model generated is very crude and lose almost all constraints >> and >> > all inheritance features (if any), because it focuses on the syntax and >> is >> > essentially converted from an JSON object compliant with a protocol >> instead >> > of from the protocol itself. Hence this result is more useful in >> > determining which JSON based protocols cannot have a syntactically >> > equivalent YANG model, than in generating a good YANG model. >> > >> > 3.3. Conclusion >> > >> > Our claim holds. A JSON based protocol carried by HTTP can have >> > syntactically equivalent YANG model, if and only if all the keys in the >> > key-value pairs in the JSON message are pre-defined keywords. >> > >> > 4. Semantic equivalence >> > >> > For JSON based protocols that don’t satisfy the message encoding >> condition, >> > it is still possible to have a semantically equivalent YANG model. All >> that >> > is required for the protocol compliant clients and the YANG model >> compliant >> > server to interoperate is an adapter which does the following: >> > >> > 1) translate FROM YANG server compliant response msg TO alto compliant >> > response msg >> > >> > 2) translate FROM alto compliant request msg TO YANG server compliant >> > request msg >> > >> > 4.1. Claim >> > >> > This adapter needs to be protocol-aware. >> > >> > Ideally, given any YANG model, we would like to be able to automatically >> > (or at least mechanically) generate this message adapter, which means >> not >> > looking at the protocol or its compliant msgs. However, without knowing >> the >> > specific protocol that we are working with (i.e. human intervention, >> i.e. >> > looking at the protocol compliant msgs), such an adapter cannot be >> > auto-generated. >> > >> > 4.2. Proof by Indistinguishability >> > >> > Suppose both the YANG server compliant msg m_y and the actually protocol >> > compliant msg m_p are in JSON (or have been encoded into JSON). Looking >> at >> > the differences between the two messages, call these differences {d1, >> d2, >> > ..., dn}. The goal for the auto-generated adapter would be to identify >> and >> > eliminate these differences. Construct a new JSON msg m' where all but >> one >> > difference di is the same as m_p and di is the same as the m_y. Without >> > looking at the protocol (or m_p), the auto-generated adapter would not >> be >> > able to distinguish between m' and m_p in its translation process, which >> > means, it won't be able to tell whether it should change di or not. >> Hence, >> > such an adapter must be protocol-aware. >> > >> > A good example is the dependent-vtag in the ALTO protocol: >> > >> > "dependent-vtag" : [ >> > >> > { >> > >> > "resource-id" : "my-network-map", >> > >> > "tag" : "abcd1234" >> > >> > } >> > >> > ] >> > >> > It was specified this way in the alto protocol. However, it could >> > conceivably be the case that it was originally the following map >> structure, >> > and was converted into the above encoding because of the map->list+key >> > issue. (This case is actually one of the few differences in the m_y and >> m_p >> > where the adapter does not need to convert it back to a map structure.) >> > >> > "dependent-vtag" : { >> > >> > "my-network-map" : { >> > >> > "tag" : "abcd1234" >> > >> > } >> > >> > } >> > >> > Without knowing the protocol, there is no way to tell. >> > >> > 5. Ramifications >> > >> > We now understand the basic condition for a JSON based protocol to have >> a >> > YANG Model. For the protocols that don’t meet this condition, there can >> be >> > a semantic equivalent YANG model, but there won’t be a generic process >> of >> > generating the adapter for all such protocols. >> > _______________________________________________ >> > netmod mailing list >> > [email protected] >> > https://www.ietf.org/mailman/listinfo/netmod >> >> -- >> Ladislav Lhotka, CZ.NIC Labs >> PGP Key ID: E74E8C0C >> >> _______________________________________________ >> netmod mailing list >> [email protected] >> https://www.ietf.org/mailman/listinfo/netmod >> > >
_______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
