Xiao SHI <[email protected]> writes: > Hi folks, > > > As a few of us were working on modeling the ALTO protocol using YANG, we > were pondering on a more general question: can YANG model all JSON based > protocols? What is the condition for a JSON based protocol (at least the > message format) to have an syntactically equivalent, hence interoperable, > YANG model with JSON encoding? Alternatively, in order to interoperate, > semantic equivalence might be sufficient, is there any condition for > semantic equivalence? > > > tl;dr: A JSON based protocol carried by HTTP can have syntactically > equivalent YANG model, if and only if all the keys in the key-value pairs > in the JSON message are pre-defined keywords in the protocol. >
Generating a YANG data model from existing instance data seems backwards to me, although Examplotron (http://examplotron.org) does something quite similar. In any case, it might be quite challenging because YANG isn't intended as a general schema language. Your observation above is true but it is just one aspect of the problem. For instance "foo": [ { "bar": 1 }, 2 ] or "foo": null cannot be modelled in YANG using the encoding of draft-ietf-netmod-yang-json. Lada > > We've come up with a few ideas, and a few things below are work in > progress, but we would love your feedback! > > > Thank you! > > Xiao > > > ====================== > > 1. Introduction. > > JavaScript Object Notation (JSON) has been a popular choice as the message > encoding for many network protocols such as the Application Layer Traffic > Optimization (ALTO) protocol, the Content Delivery Networks Interconnection > (CDNi) protocol, etc. > > Meanwhile, there are broad interests in the networking community to use > YANG to define data model so that one can use YANG related tools such as > OpenDayLight controller. Although YANG itself is XML based, there have been > efforts to model JSON content using YANG [draft-ietf-netmod-yang-json-01]. > > A natural question rises: can YANG model all JSON based protocols? What is > the condition for a JSON based protocol (at least the message format) to > have an syntactically equivalent, hence interoperable, YANG model with JSON > encoding? Alternatively, in order to interoperate, semantic equivalence > might be sufficient, is there any condition for semantic equivalence? > > 2. Claim > > A JSON based protocol carried by HTTP can have syntactically equivalent > YANG model, if and only if: > > (1) the message encoding condition is met; > > (2) the uri condition is met. > > 2.1. The message encoding condition > > The JSON message encoding MUST not contain a variable as a key in a JSON > object key-value pair. In other words, the keys in the JSON message must be > an already defined constant (or keyword) in the message format of the > protocol. > > For example, if I have a protocol where “network-map”, “src”, and “dsts” > are defined keywords, JSON text A meets the condition whereas B does not > because “PID1” and “PID2” are not protocol keywords (they are resource IDs). > > A: > > { > > “network-map”: [ > > { > > “src”: “PID1”, > > “dsts”: [“PID2”, “PID3”] > > }, > > { > > “src”: “PID2”, > > “dsts”: [“PID3”, “PID4”] > > } > > ] > > } > > B: > > > { > > “network-map”: [ > > “PID1: [“PID2”, “PID3”], > > “PID2”: [“PID3”, “PID4”] > > ] > > } > >>From this we know that since ALTO protocol uses encoding B, there cannot be > a syntactically equivalent YANG model. > > 2.2 The URI condition > > Some of the YANG-related protocols might have URI constraints, e.g. > RESTCONF. For now, we assume either that the JSON-based protocol URI could > be conformed to RESTCONF compliant uri, or that the server could have a > routing mapping between the protocol compliant uri and the RESTCONF > compliant uri, hence this condition would not be an issue, which allows us > to focus on the message encoding condition. > > 3. Proof > > 3.1. The message encoding condition is necessary. > > We first note that this condition is a necessary condition for a JSON based > protocol to have a syntactically equivalent YANG model by proving its > contrapositive. > > If one of the keys in the key-value pair in the JSON document is not > pre-defined, the corresponding XML tags will not be pre-defined keywords. > Therefore, it would not possible to model it in YANG without using > YANG’s anyxml > statement (which allows arbitrary XML content). > > However, using the anyxml statement would defeat our purpose of modeling > the data as it allows arbitrary XML content, and will not be helpful in the > subsequent parsing process. > > 3.2. The message encoding condition is sufficient. > > We prove this by providing a translation procedure from a JSON message that > is compliant with the protocol we are trying to model, to a custom java > class that can be used for Jackson data binding, then to a YANG model. > > We note that the middle step of translating to and from the custom parser > class is not necessary, but it will be useful. > > 3.2.1. motivation > > JSON data binding is the process of binding data structures (objects, > arrays, etc.) in JSON to the appropriate data structures in the server > (e.g. java classes, database tables, etc.) in parsing the JSON text. In > order to process JSON messages in a meaningful manner, data binding is > necessary. Even if the binding is not explicit, the server would need to do > it eventually. For example, one can read JSON content in a stream without > binding it to the java classes, but eventually in order to make sense of > the data, the server would eventually have to organize it, which is roughly > analogous to data binding upfront. Popular choices for JSON parsing and > data binding include jackson and gson. > > We use Jackson full data binding as our approach. Full data binding binds > JSON content into plain old java objects (POJOs), i.e. this custom parser > class can neither extend nor implement any other class. Jackson uses > ObjectMapper with the custom parser class to parse JSON content into this > class. > > 3.2.2. The message encoding condition > > The message encoding condition is that all keys in each key-value pair in > the JSON text must be pre-defined keywords. As the keys will become either > class names and instance variable names, or be keys in the java maps, it is > easy to see that the condition is equivalent to “there exists a full data > binding in Jackson custom parser class without using any map structures > (Map<String, ?>).” > > 3.2.3. Proof > > We provide a recursive binding process from a JSON object to the Jackson > custom parser class to be used by Jackson ObjectMapper. > > Type determine_type(value) { > > if (value is of a primitive type, i.e. string, number, boolean, null) { > > return the corresponding java primitive type; > > } > > if (value is a JSON object) { > > return build_parser_class(value).class; > > } > > if (value is an array) { > > return ArrayList<T> where T=determine_type(value[0]); > > } > > // should not reach here. > > } > > Class build_parser_class(JSONObject obj) { > > create custom class C; > > for each key/value pair in the obj { > > add instance variable v in C; > > the name of variable v <- key; // (__known__ a/c to our assumption) > > the type of variable v <- determine_type(value); > > } > > return C.class; > > } > > Naming: > > --change everything into CamelCase (i.e. remove dashes, etc.) > > --for instance variables, use “my” prefix, (e.g. myVariable, myNetworkMap, > etc.) > > --for the custom class name, if the object is an element of the array, use > “Element” suffix. > > This is just one convention so that the next step proceeds smoothly. As > long as this naming translation is consistent with the naming stage in the > next step, it will work just fine. > > Example (a modified ALTO protocol network map example): > > JSON object: > > { > > “meta”: { > > “resource-id”: “my-default-map”, > > “tag”: “aab875ef69c87d012” > > }, > > “network-map”: [ > > { > > “src”: “PID1”, > > “dsts”: [“PID1”, “PID2”, “PID3”] > > }, > > { > > “src”: “PID2”, > > “dsts”: [“PID1”, “PID3”] > > }, > > { > > “src”: “PID3”, > > “dsts”: [“PID2”, “PID3”] > > } > > ] > > } > > Result of build_parser_class(obj): > > Class JSONObject { > > Meta myMeta; > > ArrayList<NetworkMapElement> myNetworkMap; > > } > > Class Meta { > > String myResourceId; > > String myTag; > > } > > Class NetworkMapElement { > > String mySrc; > > ArrayList<String> myDsts; > > } > > Now given the Jackson Parser Java Class, to get a syntactically equivalent > YANG model: > > YANGModel build_yang_model(Class C) { > > for each instance variable (Type, Name) { > > if (Type is primitive type: string, number, boolean, null) { > > add the following to the YANG module: > > “leaf Name { type <YANG equivalent of Type>; }” > > } > > if (Type is an ArrayList<TypeElement>) { > > if (TypeElement is primitive type) { > > add the following to the YANG module: > > “leaf-list Name { type <YANG equivalent of TypeElement>; }” > > } else { > > // TypeElement is a custom parser class > > add the following to the YANG module: > > “list Name { <result from build_yang_model<TypeElement.class>> }” > > } > > } > > if (Type is a custom parser class) { > > add the following to the YANG module: > > “container Name { <result from build_yang_model<Type.class>> }” > > } > > } > > } > > Result from the previous example: > > container meta { > > leaf resource-id { > > type string; > > } > > leaf tag { > > type string; > > } > > } > > list network-map { > > leaf src { > > type string; > > } > > leaf-list dsts { > > type string; > > } > > } > > This does validate the JSON document with XML-JSON encoding. For your > reference, this is the XML document which validates: > > <?xml version="1.0" encoding="UTF-8" ?> > > <meta> > > <resource-id>my-default-map</resource-id> > > <tag>aab875ef69c87d012</tag> > > </meta> > > <network-map> > > <src>PID1</src> > > <dsts>PID1</dsts> > > <dsts>PID2</dsts> > > <dsts>PID3</dsts> > > </network-map> > > <network-map> > > <src>PID2</src> > > <dsts>PID1</dsts> > > <dsts>PID3</dsts> > > </network-map> > > <network-map> > > <src>PID3</src> > > <dsts>PID2</dsts> > > <dsts>PID3</dsts> > > </network-map> > > This proves that the message encoding condition is a sufficient condition > for the JSON object to have a YANG model. > > Note the model generated is very crude and lose almost all constraints and > all inheritance features (if any), because it focuses on the syntax and is > essentially converted from an JSON object compliant with a protocol instead > of from the protocol itself. Hence this result is more useful in > determining which JSON based protocols cannot have a syntactically > equivalent YANG model, than in generating a good YANG model. > > 3.3. Conclusion > > Our claim holds. A JSON based protocol carried by HTTP can have > syntactically equivalent YANG model, if and only if all the keys in the > key-value pairs in the JSON message are pre-defined keywords. > > 4. Semantic equivalence > > For JSON based protocols that don’t satisfy the message encoding condition, > it is still possible to have a semantically equivalent YANG model. All that > is required for the protocol compliant clients and the YANG model compliant > server to interoperate is an adapter which does the following: > > 1) translate FROM YANG server compliant response msg TO alto compliant > response msg > > 2) translate FROM alto compliant request msg TO YANG server compliant > request msg > > 4.1. Claim > > This adapter needs to be protocol-aware. > > Ideally, given any YANG model, we would like to be able to automatically > (or at least mechanically) generate this message adapter, which means not > looking at the protocol or its compliant msgs. However, without knowing the > specific protocol that we are working with (i.e. human intervention, i.e. > looking at the protocol compliant msgs), such an adapter cannot be > auto-generated. > > 4.2. Proof by Indistinguishability > > Suppose both the YANG server compliant msg m_y and the actually protocol > compliant msg m_p are in JSON (or have been encoded into JSON). Looking at > the differences between the two messages, call these differences {d1, d2, > ..., dn}. The goal for the auto-generated adapter would be to identify and > eliminate these differences. Construct a new JSON msg m' where all but one > difference di is the same as m_p and di is the same as the m_y. Without > looking at the protocol (or m_p), the auto-generated adapter would not be > able to distinguish between m' and m_p in its translation process, which > means, it won't be able to tell whether it should change di or not. Hence, > such an adapter must be protocol-aware. > > A good example is the dependent-vtag in the ALTO protocol: > > "dependent-vtag" : [ > > { > > "resource-id" : "my-network-map", > > "tag" : "abcd1234" > > } > > ] > > It was specified this way in the alto protocol. However, it could > conceivably be the case that it was originally the following map structure, > and was converted into the above encoding because of the map->list+key > issue. (This case is actually one of the few differences in the m_y and m_p > where the adapter does not need to convert it back to a map structure.) > > "dependent-vtag" : { > > "my-network-map" : { > > "tag" : "abcd1234" > > } > > } > > Without knowing the protocol, there is no way to tell. > > 5. Ramifications > > We now understand the basic condition for a JSON based protocol to have a > YANG Model. For the protocols that don’t meet this condition, there can be > a semantic equivalent YANG model, but there won’t be a generic process of > generating the adapter for all such protocols. > _______________________________________________ > netmod mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/netmod -- Ladislav Lhotka, CZ.NIC Labs PGP Key ID: E74E8C0C _______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
