On Mon, Oct 20, 2014 at 8:33 AM, Xiao SHI <[email protected]> wrote: > Hi Andy, > > I agree with you. Just to clarify, what I am trying to do is to use YANG > data structures for the protocol whose message encoding is JSON. I am not > trying to use JSON as a data structure. Does this make sense? > > yes -- so you want to write the YANG module that corresponds to the existing JSON usage. Not extend YANG to support all possible JSON usage. Good.
> Best, > Xiao > Andy > > On Mon, Oct 20, 2014 at 11:29 AM, Andy Bierman <[email protected]> wrote: > >> >> >> On Mon, Oct 20, 2014 at 6:23 AM, Ladislav Lhotka <[email protected]> wrote: >> >>> Xiao SHI <[email protected]> writes: >>> >>> > Hi folks, >>> > >>> > >>> > As a few of us were working on modeling the ALTO protocol using YANG, >>> we >>> > were pondering on a more general question: can YANG model all JSON >>> based >>> > protocols? What is the condition for a JSON based protocol (at least >>> the >>> > message format) to have an syntactically equivalent, hence >>> interoperable, >>> > YANG model with JSON encoding? Alternatively, in order to interoperate, >>> > semantic equivalence might be sufficient, is there any condition for >>> > semantic equivalence? >>> > >>> > >>> > tl;dr: A JSON based protocol carried by HTTP can have syntactically >>> > equivalent YANG model, if and only if all the keys in the key-value >>> pairs >>> > in the JSON message are pre-defined keywords in the protocol. >>> > >>> >>> Generating a YANG data model from existing instance data seems backwards >>> to me, although Examplotron (http://examplotron.org) does something >>> quite >>> similar. In any case, it might be quite challenging because YANG isn't >>> intended as a general schema language. Your observation above is true >>> but it is just one aspect of the problem. For instance >>> >>> "foo": [ { "bar": 1 }, 2 ] >>> >>> or >>> >>> "foo": null >>> >>> cannot be modelled in YANG using the encoding of >>> draft-ietf-netmod-yang-json. >>> >>> >> >> I think perhaps RESTCONF is overloading JSON and this has led to some >> confusion. >> RESTCONF uses JSON strictly as an encoding format, not a data structure. >> JSON is just >> the message encoding. The corresponding instance document must conform to >> the YANG definitions (not just be valid JSON). >> >> A protocol should pick between YANG data structures and JSON data >> structures, >> and not try to do both. >> >> >> >> >>> Lada >>> >>> >> Andy >> >> >>> > >>> > We've come up with a few ideas, and a few things below are work in >>> > progress, but we would love your feedback! >>> > >>> > >>> > Thank you! >>> > >>> > Xiao >>> > >>> > >>> > ====================== >>> > >>> > 1. Introduction. >>> > >>> > JavaScript Object Notation (JSON) has been a popular choice as the >>> message >>> > encoding for many network protocols such as the Application Layer >>> Traffic >>> > Optimization (ALTO) protocol, the Content Delivery Networks >>> Interconnection >>> > (CDNi) protocol, etc. >>> > >>> > Meanwhile, there are broad interests in the networking community to use >>> > YANG to define data model so that one can use YANG related tools such >>> as >>> > OpenDayLight controller. Although YANG itself is XML based, there have >>> been >>> > efforts to model JSON content using YANG >>> [draft-ietf-netmod-yang-json-01]. >>> > >>> > A natural question rises: can YANG model all JSON based protocols? >>> What is >>> > the condition for a JSON based protocol (at least the message format) >>> to >>> > have an syntactically equivalent, hence interoperable, YANG model with >>> JSON >>> > encoding? Alternatively, in order to interoperate, semantic equivalence >>> > might be sufficient, is there any condition for semantic equivalence? >>> > >>> > 2. Claim >>> > >>> > A JSON based protocol carried by HTTP can have syntactically equivalent >>> > YANG model, if and only if: >>> > >>> > (1) the message encoding condition is met; >>> > >>> > (2) the uri condition is met. >>> > >>> > 2.1. The message encoding condition >>> > >>> > The JSON message encoding MUST not contain a variable as a key in a >>> JSON >>> > object key-value pair. In other words, the keys in the JSON message >>> must be >>> > an already defined constant (or keyword) in the message format of the >>> > protocol. >>> > >>> > For example, if I have a protocol where "network-map", "src", and >>> "dsts" >>> > are defined keywords, JSON text A meets the condition whereas B does >>> not >>> > because "PID1" and "PID2" are not protocol keywords (they are resource >>> IDs). >>> > >>> > A: >>> > >>> > { >>> > >>> > "network-map": [ >>> > >>> > { >>> > >>> > "src": "PID1", >>> > >>> > "dsts": ["PID2", "PID3"] >>> > >>> > }, >>> > >>> > { >>> > >>> > "src": "PID2", >>> > >>> > "dsts": ["PID3", "PID4"] >>> > >>> > } >>> > >>> > ] >>> > >>> > } >>> > >>> > B: >>> > >>> > >>> > { >>> > >>> > "network-map": [ >>> > >>> > "PID1: ["PID2", "PID3"], >>> > >>> > "PID2": ["PID3", "PID4"] >>> > >>> > ] >>> > >>> > } >>> > >>> >>From this we know that since ALTO protocol uses encoding B, there >>> cannot be >>> > a syntactically equivalent YANG model. >>> > >>> > 2.2 The URI condition >>> > >>> > Some of the YANG-related protocols might have URI constraints, e.g. >>> > RESTCONF. For now, we assume either that the JSON-based protocol URI >>> could >>> > be conformed to RESTCONF compliant uri, or that the server could have a >>> > routing mapping between the protocol compliant uri and the RESTCONF >>> > compliant uri, hence this condition would not be an issue, which >>> allows us >>> > to focus on the message encoding condition. >>> > >>> > 3. Proof >>> > >>> > 3.1. The message encoding condition is necessary. >>> > >>> > We first note that this condition is a necessary condition for a JSON >>> based >>> > protocol to have a syntactically equivalent YANG model by proving its >>> > contrapositive. >>> > >>> > If one of the keys in the key-value pair in the JSON document is not >>> > pre-defined, the corresponding XML tags will not be pre-defined >>> keywords. >>> > Therefore, it would not possible to model it in YANG without using >>> > YANG's anyxml >>> > statement (which allows arbitrary XML content). >>> > >>> > However, using the anyxml statement would defeat our purpose of >>> modeling >>> > the data as it allows arbitrary XML content, and will not be helpful >>> in the >>> > subsequent parsing process. >>> > >>> > 3.2. The message encoding condition is sufficient. >>> > >>> > We prove this by providing a translation procedure from a JSON message >>> that >>> > is compliant with the protocol we are trying to model, to a custom java >>> > class that can be used for Jackson data binding, then to a YANG model. >>> > >>> > We note that the middle step of translating to and from the custom >>> parser >>> > class is not necessary, but it will be useful. >>> > >>> > 3.2.1. motivation >>> > >>> > JSON data binding is the process of binding data structures (objects, >>> > arrays, etc.) in JSON to the appropriate data structures in the server >>> > (e.g. java classes, database tables, etc.) in parsing the JSON text. In >>> > order to process JSON messages in a meaningful manner, data binding is >>> > necessary. Even if the binding is not explicit, the server would need >>> to do >>> > it eventually. For example, one can read JSON content in a stream >>> without >>> > binding it to the java classes, but eventually in order to make sense >>> of >>> > the data, the server would eventually have to organize it, which is >>> roughly >>> > analogous to data binding upfront. Popular choices for JSON parsing and >>> > data binding include jackson and gson. >>> > >>> > We use Jackson full data binding as our approach. Full data binding >>> binds >>> > JSON content into plain old java objects (POJOs), i.e. this custom >>> parser >>> > class can neither extend nor implement any other class. Jackson uses >>> > ObjectMapper with the custom parser class to parse JSON content into >>> this >>> > class. >>> > >>> > 3.2.2. The message encoding condition >>> > >>> > The message encoding condition is that all keys in each key-value pair >>> in >>> > the JSON text must be pre-defined keywords. As the keys will become >>> either >>> > class names and instance variable names, or be keys in the java maps, >>> it is >>> > easy to see that the condition is equivalent to "there exists a full >>> data >>> > binding in Jackson custom parser class without using any map structures >>> > (Map<String, ?>)." >>> > >>> > 3.2.3. Proof >>> > >>> > We provide a recursive binding process from a JSON object to the >>> Jackson >>> > custom parser class to be used by Jackson ObjectMapper. >>> > >>> > Type determine_type(value) { >>> > >>> > if (value is of a primitive type, i.e. string, number, boolean, null) >>> { >>> > >>> > return the corresponding java primitive type; >>> > >>> > } >>> > >>> > if (value is a JSON object) { >>> > >>> > return build_parser_class(value).class; >>> > >>> > } >>> > >>> > if (value is an array) { >>> > >>> > return ArrayList<T> where T=determine_type(value[0]); >>> > >>> > } >>> > >>> > // should not reach here. >>> > >>> > } >>> > >>> > Class build_parser_class(JSONObject obj) { >>> > >>> > create custom class C; >>> > >>> > for each key/value pair in the obj { >>> > >>> > add instance variable v in C; >>> > >>> > the name of variable v <- key; // (__known__ a/c to our assumption) >>> > >>> > the type of variable v <- determine_type(value); >>> > >>> > } >>> > >>> > return C.class; >>> > >>> > } >>> > >>> > Naming: >>> > >>> > --change everything into CamelCase (i.e. remove dashes, etc.) >>> > >>> > --for instance variables, use "my" prefix, (e.g. myVariable, >>> myNetworkMap, >>> > etc.) >>> > >>> > --for the custom class name, if the object is an element of the array, >>> use >>> > "Element" suffix. >>> > >>> > This is just one convention so that the next step proceeds smoothly. As >>> > long as this naming translation is consistent with the naming stage in >>> the >>> > next step, it will work just fine. >>> > >>> > Example (a modified ALTO protocol network map example): >>> > >>> > JSON object: >>> > >>> > { >>> > >>> > "meta": { >>> > >>> > "resource-id": "my-default-map", >>> > >>> > "tag": "aab875ef69c87d012" >>> > >>> > }, >>> > >>> > "network-map": [ >>> > >>> > { >>> > >>> > "src": "PID1", >>> > >>> > "dsts": ["PID1", "PID2", "PID3"] >>> > >>> > }, >>> > >>> > { >>> > >>> > "src": "PID2", >>> > >>> > "dsts": ["PID1", "PID3"] >>> > >>> > }, >>> > >>> > { >>> > >>> > "src": "PID3", >>> > >>> > "dsts": ["PID2", "PID3"] >>> > >>> > } >>> > >>> > ] >>> > >>> > } >>> > >>> > Result of build_parser_class(obj): >>> > >>> > Class JSONObject { >>> > >>> > Meta myMeta; >>> > >>> > ArrayList<NetworkMapElement> myNetworkMap; >>> > >>> > } >>> > >>> > Class Meta { >>> > >>> > String myResourceId; >>> > >>> > String myTag; >>> > >>> > } >>> > >>> > Class NetworkMapElement { >>> > >>> > String mySrc; >>> > >>> > ArrayList<String> myDsts; >>> > >>> > } >>> > >>> > Now given the Jackson Parser Java Class, to get a syntactically >>> equivalent >>> > YANG model: >>> > >>> > YANGModel build_yang_model(Class C) { >>> > >>> > for each instance variable (Type, Name) { >>> > >>> > if (Type is primitive type: string, number, boolean, null) { >>> > >>> > add the following to the YANG module: >>> > >>> > "leaf Name { type <YANG equivalent of Type>; }" >>> > >>> > } >>> > >>> > if (Type is an ArrayList<TypeElement>) { >>> > >>> > if (TypeElement is primitive type) { >>> > >>> > add the following to the YANG module: >>> > >>> > "leaf-list Name { type <YANG equivalent of TypeElement>; }" >>> > >>> > } else { >>> > >>> > // TypeElement is a custom parser class >>> > >>> > add the following to the YANG module: >>> > >>> > "list Name { <result from build_yang_model<TypeElement.class>> >>> }" >>> > >>> > } >>> > >>> > } >>> > >>> > if (Type is a custom parser class) { >>> > >>> > add the following to the YANG module: >>> > >>> > "container Name { <result from build_yang_model<Type.class>> }" >>> > >>> > } >>> > >>> > } >>> > >>> > } >>> > >>> > Result from the previous example: >>> > >>> > container meta { >>> > >>> > leaf resource-id { >>> > >>> > type string; >>> > >>> > } >>> > >>> > leaf tag { >>> > >>> > type string; >>> > >>> > } >>> > >>> > } >>> > >>> > list network-map { >>> > >>> > leaf src { >>> > >>> > type string; >>> > >>> > } >>> > >>> > leaf-list dsts { >>> > >>> > type string; >>> > >>> > } >>> > >>> > } >>> > >>> > This does validate the JSON document with XML-JSON encoding. For your >>> > reference, this is the XML document which validates: >>> > >>> > <?xml version="1.0" encoding="UTF-8" ?> >>> > >>> > <meta> >>> > >>> > <resource-id>my-default-map</resource-id> >>> > >>> > <tag>aab875ef69c87d012</tag> >>> > >>> > </meta> >>> > >>> > <network-map> >>> > >>> > <src>PID1</src> >>> > >>> > <dsts>PID1</dsts> >>> > >>> > <dsts>PID2</dsts> >>> > >>> > <dsts>PID3</dsts> >>> > >>> > </network-map> >>> > >>> > <network-map> >>> > >>> > <src>PID2</src> >>> > >>> > <dsts>PID1</dsts> >>> > >>> > <dsts>PID3</dsts> >>> > >>> > </network-map> >>> > >>> > <network-map> >>> > >>> > <src>PID3</src> >>> > >>> > <dsts>PID2</dsts> >>> > >>> > <dsts>PID3</dsts> >>> > >>> > </network-map> >>> > >>> > This proves that the message encoding condition is a sufficient >>> condition >>> > for the JSON object to have a YANG model. >>> > >>> > Note the model generated is very crude and lose almost all constraints >>> and >>> > all inheritance features (if any), because it focuses on the syntax >>> and is >>> > essentially converted from an JSON object compliant with a protocol >>> instead >>> > of from the protocol itself. Hence this result is more useful in >>> > determining which JSON based protocols cannot have a syntactically >>> > equivalent YANG model, than in generating a good YANG model. >>> > >>> > 3.3. Conclusion >>> > >>> > Our claim holds. A JSON based protocol carried by HTTP can have >>> > syntactically equivalent YANG model, if and only if all the keys in the >>> > key-value pairs in the JSON message are pre-defined keywords. >>> > >>> > 4. Semantic equivalence >>> > >>> > For JSON based protocols that don't satisfy the message encoding >>> condition, >>> > it is still possible to have a semantically equivalent YANG model. All >>> that >>> > is required for the protocol compliant clients and the YANG model >>> compliant >>> > server to interoperate is an adapter which does the following: >>> > >>> > 1) translate FROM YANG server compliant response msg TO alto compliant >>> > response msg >>> > >>> > 2) translate FROM alto compliant request msg TO YANG server compliant >>> > request msg >>> > >>> > 4.1. Claim >>> > >>> > This adapter needs to be protocol-aware. >>> > >>> > Ideally, given any YANG model, we would like to be able to >>> automatically >>> > (or at least mechanically) generate this message adapter, which means >>> not >>> > looking at the protocol or its compliant msgs. However, without >>> knowing the >>> > specific protocol that we are working with (i.e. human intervention, >>> i.e. >>> > looking at the protocol compliant msgs), such an adapter cannot be >>> > auto-generated. >>> > >>> > 4.2. Proof by Indistinguishability >>> > >>> > Suppose both the YANG server compliant msg m_y and the actually >>> protocol >>> > compliant msg m_p are in JSON (or have been encoded into JSON). >>> Looking at >>> > the differences between the two messages, call these differences {d1, >>> d2, >>> > ..., dn}. The goal for the auto-generated adapter would be to identify >>> and >>> > eliminate these differences. Construct a new JSON msg m' where all but >>> one >>> > difference di is the same as m_p and di is the same as the m_y. Without >>> > looking at the protocol (or m_p), the auto-generated adapter would not >>> be >>> > able to distinguish between m' and m_p in its translation process, >>> which >>> > means, it won't be able to tell whether it should change di or not. >>> Hence, >>> > such an adapter must be protocol-aware. >>> > >>> > A good example is the dependent-vtag in the ALTO protocol: >>> > >>> > "dependent-vtag" : [ >>> > >>> > { >>> > >>> > "resource-id" : "my-network-map", >>> > >>> > "tag" : "abcd1234" >>> > >>> > } >>> > >>> > ] >>> > >>> > It was specified this way in the alto protocol. However, it could >>> > conceivably be the case that it was originally the following map >>> structure, >>> > and was converted into the above encoding because of the map->list+key >>> > issue. (This case is actually one of the few differences in the m_y >>> and m_p >>> > where the adapter does not need to convert it back to a map structure.) >>> > >>> > "dependent-vtag" : { >>> > >>> > "my-network-map" : { >>> > >>> > "tag" : "abcd1234" >>> > >>> > } >>> > >>> > } >>> > >>> > Without knowing the protocol, there is no way to tell. >>> > >>> > 5. Ramifications >>> > >>> > We now understand the basic condition for a JSON based protocol to >>> have a >>> > YANG Model. For the protocols that don't meet this condition, there >>> can be >>> > a semantic equivalent YANG model, but there won't be a generic process >>> of >>> > generating the adapter for all such protocols. >>> > _______________________________________________ >>> > netmod mailing list >>> > [email protected] >>> > https://www.ietf.org/mailman/listinfo/netmod >>> >>> -- >>> Ladislav Lhotka, CZ.NIC Labs >>> PGP Key ID: E74E8C0C >>> >>> _______________________________________________ >>> netmod mailing list >>> [email protected] >>> https://www.ietf.org/mailman/listinfo/netmod >>> >> >> >
_______________________________________________ alto mailing list [email protected] https://www.ietf.org/mailman/listinfo/alto
