On Mon, Oct 20, 2014 at 6:23 AM, Ladislav Lhotka <[email protected]> wrote:
> Xiao SHI <[email protected]> writes: > > > Hi folks, > > > > > > As a few of us were working on modeling the ALTO protocol using YANG, we > > were pondering on a more general question: can YANG model all JSON based > > protocols? What is the condition for a JSON based protocol (at least the > > message format) to have an syntactically equivalent, hence interoperable, > > YANG model with JSON encoding? Alternatively, in order to interoperate, > > semantic equivalence might be sufficient, is there any condition for > > semantic equivalence? > > > > > > tl;dr: A JSON based protocol carried by HTTP can have syntactically > > equivalent YANG model, if and only if all the keys in the key-value pairs > > in the JSON message are pre-defined keywords in the protocol. > > > > Generating a YANG data model from existing instance data seems backwards > to me, although Examplotron (http://examplotron.org) does something quite > similar. In any case, it might be quite challenging because YANG isn't > intended as a general schema language. Your observation above is true > but it is just one aspect of the problem. For instance > > "foo": [ { "bar": 1 }, 2 ] > > or > > "foo": null > > cannot be modelled in YANG using the encoding of > draft-ietf-netmod-yang-json. > > I think perhaps RESTCONF is overloading JSON and this has led to some confusion. RESTCONF uses JSON strictly as an encoding format, not a data structure. JSON is just the message encoding. The corresponding instance document must conform to the YANG definitions (not just be valid JSON). A protocol should pick between YANG data structures and JSON data structures, and not try to do both. > Lada > > Andy > > > > We've come up with a few ideas, and a few things below are work in > > progress, but we would love your feedback! > > > > > > Thank you! > > > > Xiao > > > > > > ====================== > > > > 1. Introduction. > > > > JavaScript Object Notation (JSON) has been a popular choice as the > message > > encoding for many network protocols such as the Application Layer Traffic > > Optimization (ALTO) protocol, the Content Delivery Networks > Interconnection > > (CDNi) protocol, etc. > > > > Meanwhile, there are broad interests in the networking community to use > > YANG to define data model so that one can use YANG related tools such as > > OpenDayLight controller. Although YANG itself is XML based, there have > been > > efforts to model JSON content using YANG > [draft-ietf-netmod-yang-json-01]. > > > > A natural question rises: can YANG model all JSON based protocols? What > is > > the condition for a JSON based protocol (at least the message format) to > > have an syntactically equivalent, hence interoperable, YANG model with > JSON > > encoding? Alternatively, in order to interoperate, semantic equivalence > > might be sufficient, is there any condition for semantic equivalence? > > > > 2. Claim > > > > A JSON based protocol carried by HTTP can have syntactically equivalent > > YANG model, if and only if: > > > > (1) the message encoding condition is met; > > > > (2) the uri condition is met. > > > > 2.1. The message encoding condition > > > > The JSON message encoding MUST not contain a variable as a key in a JSON > > object key-value pair. In other words, the keys in the JSON message must > be > > an already defined constant (or keyword) in the message format of the > > protocol. > > > > For example, if I have a protocol where "network-map", "src", and "dsts" > > are defined keywords, JSON text A meets the condition whereas B does not > > because "PID1" and "PID2" are not protocol keywords (they are resource > IDs). > > > > A: > > > > { > > > > "network-map": [ > > > > { > > > > "src": "PID1", > > > > "dsts": ["PID2", "PID3"] > > > > }, > > > > { > > > > "src": "PID2", > > > > "dsts": ["PID3", "PID4"] > > > > } > > > > ] > > > > } > > > > B: > > > > > > { > > > > "network-map": [ > > > > "PID1: ["PID2", "PID3"], > > > > "PID2": ["PID3", "PID4"] > > > > ] > > > > } > > > >>From this we know that since ALTO protocol uses encoding B, there cannot > be > > a syntactically equivalent YANG model. > > > > 2.2 The URI condition > > > > Some of the YANG-related protocols might have URI constraints, e.g. > > RESTCONF. For now, we assume either that the JSON-based protocol URI > could > > be conformed to RESTCONF compliant uri, or that the server could have a > > routing mapping between the protocol compliant uri and the RESTCONF > > compliant uri, hence this condition would not be an issue, which allows > us > > to focus on the message encoding condition. > > > > 3. Proof > > > > 3.1. The message encoding condition is necessary. > > > > We first note that this condition is a necessary condition for a JSON > based > > protocol to have a syntactically equivalent YANG model by proving its > > contrapositive. > > > > If one of the keys in the key-value pair in the JSON document is not > > pre-defined, the corresponding XML tags will not be pre-defined keywords. > > Therefore, it would not possible to model it in YANG without using > > YANG's anyxml > > statement (which allows arbitrary XML content). > > > > However, using the anyxml statement would defeat our purpose of modeling > > the data as it allows arbitrary XML content, and will not be helpful in > the > > subsequent parsing process. > > > > 3.2. The message encoding condition is sufficient. > > > > We prove this by providing a translation procedure from a JSON message > that > > is compliant with the protocol we are trying to model, to a custom java > > class that can be used for Jackson data binding, then to a YANG model. > > > > We note that the middle step of translating to and from the custom parser > > class is not necessary, but it will be useful. > > > > 3.2.1. motivation > > > > JSON data binding is the process of binding data structures (objects, > > arrays, etc.) in JSON to the appropriate data structures in the server > > (e.g. java classes, database tables, etc.) in parsing the JSON text. In > > order to process JSON messages in a meaningful manner, data binding is > > necessary. Even if the binding is not explicit, the server would need to > do > > it eventually. For example, one can read JSON content in a stream without > > binding it to the java classes, but eventually in order to make sense of > > the data, the server would eventually have to organize it, which is > roughly > > analogous to data binding upfront. Popular choices for JSON parsing and > > data binding include jackson and gson. > > > > We use Jackson full data binding as our approach. Full data binding binds > > JSON content into plain old java objects (POJOs), i.e. this custom parser > > class can neither extend nor implement any other class. Jackson uses > > ObjectMapper with the custom parser class to parse JSON content into this > > class. > > > > 3.2.2. The message encoding condition > > > > The message encoding condition is that all keys in each key-value pair in > > the JSON text must be pre-defined keywords. As the keys will become > either > > class names and instance variable names, or be keys in the java maps, it > is > > easy to see that the condition is equivalent to "there exists a full data > > binding in Jackson custom parser class without using any map structures > > (Map<String, ?>)." > > > > 3.2.3. Proof > > > > We provide a recursive binding process from a JSON object to the Jackson > > custom parser class to be used by Jackson ObjectMapper. > > > > Type determine_type(value) { > > > > if (value is of a primitive type, i.e. string, number, boolean, null) { > > > > return the corresponding java primitive type; > > > > } > > > > if (value is a JSON object) { > > > > return build_parser_class(value).class; > > > > } > > > > if (value is an array) { > > > > return ArrayList<T> where T=determine_type(value[0]); > > > > } > > > > // should not reach here. > > > > } > > > > Class build_parser_class(JSONObject obj) { > > > > create custom class C; > > > > for each key/value pair in the obj { > > > > add instance variable v in C; > > > > the name of variable v <- key; // (__known__ a/c to our assumption) > > > > the type of variable v <- determine_type(value); > > > > } > > > > return C.class; > > > > } > > > > Naming: > > > > --change everything into CamelCase (i.e. remove dashes, etc.) > > > > --for instance variables, use "my" prefix, (e.g. myVariable, > myNetworkMap, > > etc.) > > > > --for the custom class name, if the object is an element of the array, > use > > "Element" suffix. > > > > This is just one convention so that the next step proceeds smoothly. As > > long as this naming translation is consistent with the naming stage in > the > > next step, it will work just fine. > > > > Example (a modified ALTO protocol network map example): > > > > JSON object: > > > > { > > > > "meta": { > > > > "resource-id": "my-default-map", > > > > "tag": "aab875ef69c87d012" > > > > }, > > > > "network-map": [ > > > > { > > > > "src": "PID1", > > > > "dsts": ["PID1", "PID2", "PID3"] > > > > }, > > > > { > > > > "src": "PID2", > > > > "dsts": ["PID1", "PID3"] > > > > }, > > > > { > > > > "src": "PID3", > > > > "dsts": ["PID2", "PID3"] > > > > } > > > > ] > > > > } > > > > Result of build_parser_class(obj): > > > > Class JSONObject { > > > > Meta myMeta; > > > > ArrayList<NetworkMapElement> myNetworkMap; > > > > } > > > > Class Meta { > > > > String myResourceId; > > > > String myTag; > > > > } > > > > Class NetworkMapElement { > > > > String mySrc; > > > > ArrayList<String> myDsts; > > > > } > > > > Now given the Jackson Parser Java Class, to get a syntactically > equivalent > > YANG model: > > > > YANGModel build_yang_model(Class C) { > > > > for each instance variable (Type, Name) { > > > > if (Type is primitive type: string, number, boolean, null) { > > > > add the following to the YANG module: > > > > "leaf Name { type <YANG equivalent of Type>; }" > > > > } > > > > if (Type is an ArrayList<TypeElement>) { > > > > if (TypeElement is primitive type) { > > > > add the following to the YANG module: > > > > "leaf-list Name { type <YANG equivalent of TypeElement>; }" > > > > } else { > > > > // TypeElement is a custom parser class > > > > add the following to the YANG module: > > > > "list Name { <result from build_yang_model<TypeElement.class>> }" > > > > } > > > > } > > > > if (Type is a custom parser class) { > > > > add the following to the YANG module: > > > > "container Name { <result from build_yang_model<Type.class>> }" > > > > } > > > > } > > > > } > > > > Result from the previous example: > > > > container meta { > > > > leaf resource-id { > > > > type string; > > > > } > > > > leaf tag { > > > > type string; > > > > } > > > > } > > > > list network-map { > > > > leaf src { > > > > type string; > > > > } > > > > leaf-list dsts { > > > > type string; > > > > } > > > > } > > > > This does validate the JSON document with XML-JSON encoding. For your > > reference, this is the XML document which validates: > > > > <?xml version="1.0" encoding="UTF-8" ?> > > > > <meta> > > > > <resource-id>my-default-map</resource-id> > > > > <tag>aab875ef69c87d012</tag> > > > > </meta> > > > > <network-map> > > > > <src>PID1</src> > > > > <dsts>PID1</dsts> > > > > <dsts>PID2</dsts> > > > > <dsts>PID3</dsts> > > > > </network-map> > > > > <network-map> > > > > <src>PID2</src> > > > > <dsts>PID1</dsts> > > > > <dsts>PID3</dsts> > > > > </network-map> > > > > <network-map> > > > > <src>PID3</src> > > > > <dsts>PID2</dsts> > > > > <dsts>PID3</dsts> > > > > </network-map> > > > > This proves that the message encoding condition is a sufficient condition > > for the JSON object to have a YANG model. > > > > Note the model generated is very crude and lose almost all constraints > and > > all inheritance features (if any), because it focuses on the syntax and > is > > essentially converted from an JSON object compliant with a protocol > instead > > of from the protocol itself. Hence this result is more useful in > > determining which JSON based protocols cannot have a syntactically > > equivalent YANG model, than in generating a good YANG model. > > > > 3.3. Conclusion > > > > Our claim holds. A JSON based protocol carried by HTTP can have > > syntactically equivalent YANG model, if and only if all the keys in the > > key-value pairs in the JSON message are pre-defined keywords. > > > > 4. Semantic equivalence > > > > For JSON based protocols that don't satisfy the message encoding > condition, > > it is still possible to have a semantically equivalent YANG model. All > that > > is required for the protocol compliant clients and the YANG model > compliant > > server to interoperate is an adapter which does the following: > > > > 1) translate FROM YANG server compliant response msg TO alto compliant > > response msg > > > > 2) translate FROM alto compliant request msg TO YANG server compliant > > request msg > > > > 4.1. Claim > > > > This adapter needs to be protocol-aware. > > > > Ideally, given any YANG model, we would like to be able to automatically > > (or at least mechanically) generate this message adapter, which means not > > looking at the protocol or its compliant msgs. However, without knowing > the > > specific protocol that we are working with (i.e. human intervention, i.e. > > looking at the protocol compliant msgs), such an adapter cannot be > > auto-generated. > > > > 4.2. Proof by Indistinguishability > > > > Suppose both the YANG server compliant msg m_y and the actually protocol > > compliant msg m_p are in JSON (or have been encoded into JSON). Looking > at > > the differences between the two messages, call these differences {d1, d2, > > ..., dn}. The goal for the auto-generated adapter would be to identify > and > > eliminate these differences. Construct a new JSON msg m' where all but > one > > difference di is the same as m_p and di is the same as the m_y. Without > > looking at the protocol (or m_p), the auto-generated adapter would not be > > able to distinguish between m' and m_p in its translation process, which > > means, it won't be able to tell whether it should change di or not. > Hence, > > such an adapter must be protocol-aware. > > > > A good example is the dependent-vtag in the ALTO protocol: > > > > "dependent-vtag" : [ > > > > { > > > > "resource-id" : "my-network-map", > > > > "tag" : "abcd1234" > > > > } > > > > ] > > > > It was specified this way in the alto protocol. However, it could > > conceivably be the case that it was originally the following map > structure, > > and was converted into the above encoding because of the map->list+key > > issue. (This case is actually one of the few differences in the m_y and > m_p > > where the adapter does not need to convert it back to a map structure.) > > > > "dependent-vtag" : { > > > > "my-network-map" : { > > > > "tag" : "abcd1234" > > > > } > > > > } > > > > Without knowing the protocol, there is no way to tell. > > > > 5. Ramifications > > > > We now understand the basic condition for a JSON based protocol to have a > > YANG Model. For the protocols that don't meet this condition, there can > be > > a semantic equivalent YANG model, but there won't be a generic process of > > generating the adapter for all such protocols. > > _______________________________________________ > > netmod mailing list > > [email protected] > > https://www.ietf.org/mailman/listinfo/netmod > > -- > Ladislav Lhotka, CZ.NIC Labs > PGP Key ID: E74E8C0C > > _______________________________________________ > netmod mailing list > [email protected] > https://www.ietf.org/mailman/listinfo/netmod >
_______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
