Xiao SHI <[email protected]> writes:
> Hi Lada,
>
> Thank you so much for your feedback!
>
> The motivation here is to produce a YANG model given a defined protocol,
> e.g. ALTO, CDNi, etc. This way, we may incorporate future protocols into
> ODL controllers or use other YANG related tools to handle those protocols.
> Hence the objective of the modeling is the protocol, not the messages. I am
> not suggesting to use YANG as a schema language.
Hmm, YANG was designed for modelling datastores and contents of protocol
messages, not protocols as such.
>
> The conditions discussed in this process will be helpful in the following
> sense:
> 1) disprove the possibility to get a syntactically equivalent YANG model
> from a JSON based protocol;
> 2) generate a Jackson JSON parser class and YANG model from the protocols
> that meet the criteria;
> 3) for designers of future protocols who are thinking of using YANG related
> tools, set a guideline of what their messages should look like (or MUST not
> contain);
> 4) for the YANG/NETCONF WG, describing the compatibility issues that they
> might be interested in tackling for expanding YANG's usage.
>
> For this example, "foo": [ { "bar": 1 }, 2 ] is also not easy to do
> data-binding in the parser, because the elements of the array are not of
> the same type, which is indeed an assumption that I left out.
>
> For the {"foo”:null} example, can't we model it using type empty? That's
> equivalent to the XML element <foo /> right?
<foo/> corresponds to
"foo": [null]
Section 6.9 in draft-ietf-netmod-yang-json-01 gives the reasoning,
essentially it is because some software isn't able to distinguish
"foo": null
from "foo" being absent.
Lada
>
> Cheers,
> Xiao
>
> On Mon, Oct 20, 2014 at 9:23 AM, Ladislav Lhotka <[email protected]> wrote:
>
>> Xiao SHI <[email protected]> writes:
>>
>> > Hi folks,
>> >
>> >
>> > As a few of us were working on modeling the ALTO protocol using YANG, we
>> > were pondering on a more general question: can YANG model all JSON based
>> > protocols? What is the condition for a JSON based protocol (at least the
>> > message format) to have an syntactically equivalent, hence interoperable,
>> > YANG model with JSON encoding? Alternatively, in order to interoperate,
>> > semantic equivalence might be sufficient, is there any condition for
>> > semantic equivalence?
>> >
>> >
>> > tl;dr: A JSON based protocol carried by HTTP can have syntactically
>> > equivalent YANG model, if and only if all the keys in the key-value pairs
>> > in the JSON message are pre-defined keywords in the protocol.
>> >
>>
>> Generating a YANG data model from existing instance data seems backwards
>> to me, although Examplotron (http://examplotron.org) does something quite
>> similar. In any case, it might be quite challenging because YANG isn't
>> intended as a general schema language. Your observation above is true
>> but it is just one aspect of the problem. For instance
>>
>> "foo": [ { "bar": 1 }, 2 ]
>>
>> or
>>
>> "foo": null
>>
>> cannot be modelled in YANG using the encoding of
>> draft-ietf-netmod-yang-json.
>>
>> Lada
>>
>> >
>> > We've come up with a few ideas, and a few things below are work in
>> > progress, but we would love your feedback!
>> >
>> >
>> > Thank you!
>> >
>> > Xiao
>> >
>> >
>> > ======================
>> >
>> > 1. Introduction.
>> >
>> > JavaScript Object Notation (JSON) has been a popular choice as the
>> message
>> > encoding for many network protocols such as the Application Layer Traffic
>> > Optimization (ALTO) protocol, the Content Delivery Networks
>> Interconnection
>> > (CDNi) protocol, etc.
>> >
>> > Meanwhile, there are broad interests in the networking community to use
>> > YANG to define data model so that one can use YANG related tools such as
>> > OpenDayLight controller. Although YANG itself is XML based, there have
>> been
>> > efforts to model JSON content using YANG
>> [draft-ietf-netmod-yang-json-01].
>> >
>> > A natural question rises: can YANG model all JSON based protocols? What
>> is
>> > the condition for a JSON based protocol (at least the message format) to
>> > have an syntactically equivalent, hence interoperable, YANG model with
>> JSON
>> > encoding? Alternatively, in order to interoperate, semantic equivalence
>> > might be sufficient, is there any condition for semantic equivalence?
>> >
>> > 2. Claim
>> >
>> > A JSON based protocol carried by HTTP can have syntactically equivalent
>> > YANG model, if and only if:
>> >
>> > (1) the message encoding condition is met;
>> >
>> > (2) the uri condition is met.
>> >
>> > 2.1. The message encoding condition
>> >
>> > The JSON message encoding MUST not contain a variable as a key in a JSON
>> > object key-value pair. In other words, the keys in the JSON message must
>> be
>> > an already defined constant (or keyword) in the message format of the
>> > protocol.
>> >
>> > For example, if I have a protocol where “network-map”, “src”, and “dsts”
>> > are defined keywords, JSON text A meets the condition whereas B does not
>> > because “PID1” and “PID2” are not protocol keywords (they are resource
>> IDs).
>> >
>> > A:
>> >
>> > {
>> >
>> > “network-map”: [
>> >
>> > {
>> >
>> > “src”: “PID1”,
>> >
>> > “dsts”: [“PID2”, “PID3”]
>> >
>> > },
>> >
>> > {
>> >
>> > “src”: “PID2”,
>> >
>> > “dsts”: [“PID3”, “PID4”]
>> >
>> > }
>> >
>> > ]
>> >
>> > }
>> >
>> > B:
>> >
>> >
>> > {
>> >
>> > “network-map”: [
>> >
>> > “PID1: [“PID2”, “PID3”],
>> >
>> > “PID2”: [“PID3”, “PID4”]
>> >
>> > ]
>> >
>> > }
>> >
>> >>From this we know that since ALTO protocol uses encoding B, there cannot
>> be
>> > a syntactically equivalent YANG model.
>> >
>> > 2.2 The URI condition
>> >
>> > Some of the YANG-related protocols might have URI constraints, e.g.
>> > RESTCONF. For now, we assume either that the JSON-based protocol URI
>> could
>> > be conformed to RESTCONF compliant uri, or that the server could have a
>> > routing mapping between the protocol compliant uri and the RESTCONF
>> > compliant uri, hence this condition would not be an issue, which allows
>> us
>> > to focus on the message encoding condition.
>> >
>> > 3. Proof
>> >
>> > 3.1. The message encoding condition is necessary.
>> >
>> > We first note that this condition is a necessary condition for a JSON
>> based
>> > protocol to have a syntactically equivalent YANG model by proving its
>> > contrapositive.
>> >
>> > If one of the keys in the key-value pair in the JSON document is not
>> > pre-defined, the corresponding XML tags will not be pre-defined keywords.
>> > Therefore, it would not possible to model it in YANG without using
>> > YANG’s anyxml
>> > statement (which allows arbitrary XML content).
>> >
>> > However, using the anyxml statement would defeat our purpose of modeling
>> > the data as it allows arbitrary XML content, and will not be helpful in
>> the
>> > subsequent parsing process.
>> >
>> > 3.2. The message encoding condition is sufficient.
>> >
>> > We prove this by providing a translation procedure from a JSON message
>> that
>> > is compliant with the protocol we are trying to model, to a custom java
>> > class that can be used for Jackson data binding, then to a YANG model.
>> >
>> > We note that the middle step of translating to and from the custom parser
>> > class is not necessary, but it will be useful.
>> >
>> > 3.2.1. motivation
>> >
>> > JSON data binding is the process of binding data structures (objects,
>> > arrays, etc.) in JSON to the appropriate data structures in the server
>> > (e.g. java classes, database tables, etc.) in parsing the JSON text. In
>> > order to process JSON messages in a meaningful manner, data binding is
>> > necessary. Even if the binding is not explicit, the server would need to
>> do
>> > it eventually. For example, one can read JSON content in a stream without
>> > binding it to the java classes, but eventually in order to make sense of
>> > the data, the server would eventually have to organize it, which is
>> roughly
>> > analogous to data binding upfront. Popular choices for JSON parsing and
>> > data binding include jackson and gson.
>> >
>> > We use Jackson full data binding as our approach. Full data binding binds
>> > JSON content into plain old java objects (POJOs), i.e. this custom parser
>> > class can neither extend nor implement any other class. Jackson uses
>> > ObjectMapper with the custom parser class to parse JSON content into this
>> > class.
>> >
>> > 3.2.2. The message encoding condition
>> >
>> > The message encoding condition is that all keys in each key-value pair in
>> > the JSON text must be pre-defined keywords. As the keys will become
>> either
>> > class names and instance variable names, or be keys in the java maps, it
>> is
>> > easy to see that the condition is equivalent to “there exists a full data
>> > binding in Jackson custom parser class without using any map structures
>> > (Map<String, ?>).”
>> >
>> > 3.2.3. Proof
>> >
>> > We provide a recursive binding process from a JSON object to the Jackson
>> > custom parser class to be used by Jackson ObjectMapper.
>> >
>> > Type determine_type(value) {
>> >
>> > if (value is of a primitive type, i.e. string, number, boolean, null) {
>> >
>> > return the corresponding java primitive type;
>> >
>> > }
>> >
>> > if (value is a JSON object) {
>> >
>> > return build_parser_class(value).class;
>> >
>> > }
>> >
>> > if (value is an array) {
>> >
>> > return ArrayList<T> where T=determine_type(value[0]);
>> >
>> > }
>> >
>> > // should not reach here.
>> >
>> > }
>> >
>> > Class build_parser_class(JSONObject obj) {
>> >
>> > create custom class C;
>> >
>> > for each key/value pair in the obj {
>> >
>> > add instance variable v in C;
>> >
>> > the name of variable v <- key; // (__known__ a/c to our assumption)
>> >
>> > the type of variable v <- determine_type(value);
>> >
>> > }
>> >
>> > return C.class;
>> >
>> > }
>> >
>> > Naming:
>> >
>> > --change everything into CamelCase (i.e. remove dashes, etc.)
>> >
>> > --for instance variables, use “my” prefix, (e.g. myVariable,
>> myNetworkMap,
>> > etc.)
>> >
>> > --for the custom class name, if the object is an element of the array,
>> use
>> > “Element” suffix.
>> >
>> > This is just one convention so that the next step proceeds smoothly. As
>> > long as this naming translation is consistent with the naming stage in
>> the
>> > next step, it will work just fine.
>> >
>> > Example (a modified ALTO protocol network map example):
>> >
>> > JSON object:
>> >
>> > {
>> >
>> > “meta”: {
>> >
>> > “resource-id”: “my-default-map”,
>> >
>> > “tag”: “aab875ef69c87d012”
>> >
>> > },
>> >
>> > “network-map”: [
>> >
>> > {
>> >
>> > “src”: “PID1”,
>> >
>> > “dsts”: [“PID1”, “PID2”, “PID3”]
>> >
>> > },
>> >
>> > {
>> >
>> > “src”: “PID2”,
>> >
>> > “dsts”: [“PID1”, “PID3”]
>> >
>> > },
>> >
>> > {
>> >
>> > “src”: “PID3”,
>> >
>> > “dsts”: [“PID2”, “PID3”]
>> >
>> > }
>> >
>> > ]
>> >
>> > }
>> >
>> > Result of build_parser_class(obj):
>> >
>> > Class JSONObject {
>> >
>> > Meta myMeta;
>> >
>> > ArrayList<NetworkMapElement> myNetworkMap;
>> >
>> > }
>> >
>> > Class Meta {
>> >
>> > String myResourceId;
>> >
>> > String myTag;
>> >
>> > }
>> >
>> > Class NetworkMapElement {
>> >
>> > String mySrc;
>> >
>> > ArrayList<String> myDsts;
>> >
>> > }
>> >
>> > Now given the Jackson Parser Java Class, to get a syntactically
>> equivalent
>> > YANG model:
>> >
>> > YANGModel build_yang_model(Class C) {
>> >
>> > for each instance variable (Type, Name) {
>> >
>> > if (Type is primitive type: string, number, boolean, null) {
>> >
>> > add the following to the YANG module:
>> >
>> > “leaf Name { type <YANG equivalent of Type>; }”
>> >
>> > }
>> >
>> > if (Type is an ArrayList<TypeElement>) {
>> >
>> > if (TypeElement is primitive type) {
>> >
>> > add the following to the YANG module:
>> >
>> > “leaf-list Name { type <YANG equivalent of TypeElement>; }”
>> >
>> > } else {
>> >
>> > // TypeElement is a custom parser class
>> >
>> > add the following to the YANG module:
>> >
>> > “list Name { <result from build_yang_model<TypeElement.class>> }”
>> >
>> > }
>> >
>> > }
>> >
>> > if (Type is a custom parser class) {
>> >
>> > add the following to the YANG module:
>> >
>> > “container Name { <result from build_yang_model<Type.class>> }”
>> >
>> > }
>> >
>> > }
>> >
>> > }
>> >
>> > Result from the previous example:
>> >
>> > container meta {
>> >
>> > leaf resource-id {
>> >
>> > type string;
>> >
>> > }
>> >
>> > leaf tag {
>> >
>> > type string;
>> >
>> > }
>> >
>> > }
>> >
>> > list network-map {
>> >
>> > leaf src {
>> >
>> > type string;
>> >
>> > }
>> >
>> > leaf-list dsts {
>> >
>> > type string;
>> >
>> > }
>> >
>> > }
>> >
>> > This does validate the JSON document with XML-JSON encoding. For your
>> > reference, this is the XML document which validates:
>> >
>> > <?xml version="1.0" encoding="UTF-8" ?>
>> >
>> > <meta>
>> >
>> > <resource-id>my-default-map</resource-id>
>> >
>> > <tag>aab875ef69c87d012</tag>
>> >
>> > </meta>
>> >
>> > <network-map>
>> >
>> > <src>PID1</src>
>> >
>> > <dsts>PID1</dsts>
>> >
>> > <dsts>PID2</dsts>
>> >
>> > <dsts>PID3</dsts>
>> >
>> > </network-map>
>> >
>> > <network-map>
>> >
>> > <src>PID2</src>
>> >
>> > <dsts>PID1</dsts>
>> >
>> > <dsts>PID3</dsts>
>> >
>> > </network-map>
>> >
>> > <network-map>
>> >
>> > <src>PID3</src>
>> >
>> > <dsts>PID2</dsts>
>> >
>> > <dsts>PID3</dsts>
>> >
>> > </network-map>
>> >
>> > This proves that the message encoding condition is a sufficient condition
>> > for the JSON object to have a YANG model.
>> >
>> > Note the model generated is very crude and lose almost all constraints
>> and
>> > all inheritance features (if any), because it focuses on the syntax and
>> is
>> > essentially converted from an JSON object compliant with a protocol
>> instead
>> > of from the protocol itself. Hence this result is more useful in
>> > determining which JSON based protocols cannot have a syntactically
>> > equivalent YANG model, than in generating a good YANG model.
>> >
>> > 3.3. Conclusion
>> >
>> > Our claim holds. A JSON based protocol carried by HTTP can have
>> > syntactically equivalent YANG model, if and only if all the keys in the
>> > key-value pairs in the JSON message are pre-defined keywords.
>> >
>> > 4. Semantic equivalence
>> >
>> > For JSON based protocols that don’t satisfy the message encoding
>> condition,
>> > it is still possible to have a semantically equivalent YANG model. All
>> that
>> > is required for the protocol compliant clients and the YANG model
>> compliant
>> > server to interoperate is an adapter which does the following:
>> >
>> > 1) translate FROM YANG server compliant response msg TO alto compliant
>> > response msg
>> >
>> > 2) translate FROM alto compliant request msg TO YANG server compliant
>> > request msg
>> >
>> > 4.1. Claim
>> >
>> > This adapter needs to be protocol-aware.
>> >
>> > Ideally, given any YANG model, we would like to be able to automatically
>> > (or at least mechanically) generate this message adapter, which means not
>> > looking at the protocol or its compliant msgs. However, without knowing
>> the
>> > specific protocol that we are working with (i.e. human intervention, i.e.
>> > looking at the protocol compliant msgs), such an adapter cannot be
>> > auto-generated.
>> >
>> > 4.2. Proof by Indistinguishability
>> >
>> > Suppose both the YANG server compliant msg m_y and the actually protocol
>> > compliant msg m_p are in JSON (or have been encoded into JSON). Looking
>> at
>> > the differences between the two messages, call these differences {d1, d2,
>> > ..., dn}. The goal for the auto-generated adapter would be to identify
>> and
>> > eliminate these differences. Construct a new JSON msg m' where all but
>> one
>> > difference di is the same as m_p and di is the same as the m_y. Without
>> > looking at the protocol (or m_p), the auto-generated adapter would not be
>> > able to distinguish between m' and m_p in its translation process, which
>> > means, it won't be able to tell whether it should change di or not.
>> Hence,
>> > such an adapter must be protocol-aware.
>> >
>> > A good example is the dependent-vtag in the ALTO protocol:
>> >
>> > "dependent-vtag" : [
>> >
>> > {
>> >
>> > "resource-id" : "my-network-map",
>> >
>> > "tag" : "abcd1234"
>> >
>> > }
>> >
>> > ]
>> >
>> > It was specified this way in the alto protocol. However, it could
>> > conceivably be the case that it was originally the following map
>> structure,
>> > and was converted into the above encoding because of the map->list+key
>> > issue. (This case is actually one of the few differences in the m_y and
>> m_p
>> > where the adapter does not need to convert it back to a map structure.)
>> >
>> > "dependent-vtag" : {
>> >
>> > "my-network-map" : {
>> >
>> > "tag" : "abcd1234"
>> >
>> > }
>> >
>> > }
>> >
>> > Without knowing the protocol, there is no way to tell.
>> >
>> > 5. Ramifications
>> >
>> > We now understand the basic condition for a JSON based protocol to have a
>> > YANG Model. For the protocols that don’t meet this condition, there can
>> be
>> > a semantic equivalent YANG model, but there won’t be a generic process of
>> > generating the adapter for all such protocols.
>> > _______________________________________________
>> > netmod mailing list
>> > [email protected]
>> > https://www.ietf.org/mailman/listinfo/netmod
>>
>> --
>> Ladislav Lhotka, CZ.NIC Labs
>> PGP Key ID: E74E8C0C
>>
--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C
_______________________________________________
alto mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/alto
