On Thu, Jun 4, 2015 at 9:32 AM, Alexander Pelov <[email protected]> wrote: > Hi all, > > Just a short idea for the hash clash issue. > > The 30 bits, which are reserved for the YANG hash can have a structure : > 1 bit = structured YANG hash (0) or pure YANG hash (1) > If 0 (structured), then : > 1 bit = managed (0) or non-managed (1) > 18 bits = module ID > 10 bits = item ID > If 1 (non-structured), then: > 29 bits = least significant 29 bits of the YANG hash. >
I do not really want to change the YANG Hash from 30 bits to 29 bits. Having a different numbering scheme for every object seems too complicated. I would add the constraint that the server must use the same numbering scheme for all objects, so this can be discovered via .well-known capabilities, and all 30 bits can be used in a YANG Hash. > In case of clash, send error, the full URI must be used (which should be with > extremely low probability) > This way, in the worst case you are limiting your hash value from 30 to 29 > bits (and thus slightly increasing the clash probability). > > If structured YANG hash, then: > 0 = managed (e.g. IETF-assigned module ID) > 18 bits = the ID of the module (1, 2, 3, …., 262143) assigned > incrementally by IANA (0 reserved for special usage) > 10 bits = the ID of the item in the module, incremented at each item > (1..1023 - hope this is sufficient for all cases). > 1 = unmanaged > 18 bits = ID of module, by default the 18 less significant bits of the > module. It is up to the user to assure that there are no clashes with the > names of the other modules on the device > 10 bits = ID of the item. It is up to the implementer to ensure that > there are no clashes within the module. > > Two possible drawbacks of this approach, however: > - For managed IDs you will probably have to keep both the YANG hash and the > managed full ID. > - In case of unmanaged IDs you increase the probability of collision of > 1E-9 to 2E-9. > IMO only managed mode is needed. The IANA registry (that will be created) should be open to vendors and other SDOs. This registry just assigns a module number to a module name. The 10-bit mystery encoding does not seem very interoperable. It should be standard. I think the YANG "id" extension to number each object manually will work. > Best, > Alexander > Andy > >> Le 4 juin 2015 à 18:04, Michel Veillette <[email protected]> >> a écrit : >> >> Hi Andy, >> >> Just to be sure about the scope of your comment. >> >> You say " Each object has a YANG Hash", yes but this is not a uniquely >> object identify. >> Did we reach a consensus about using the " YANG hash" + "module name or a >> module ID" as a unique data node (object) identifier? >> >> This can take the form of a rehash table available on the CoMI server or >> elsewhere (" YANG hash", "module name or module ID", "rehash or alias") >> This can take the form of an error message containing (" YANG hash", "module >> name or module ID", "rehash or alias") >> This can take the form of a fully qualified identifier within the request or >> payload (REQ: GET example.com/mg?select(<module name>.<YANG hash>) >> >> Michel Veillette >> System Architecture Director >> Trilliant Inc. >> Tel: 450-375-0556 ext. 237 >> [email protected] >> www.trilliantinc.com >> >> >> -----Original Message----- >> From: Andy Bierman [mailto:[email protected]] >> Sent: 4 juin 2015 11:37 >> To: Michel Veillette >> Cc: Alexander Pelov; [email protected]; [email protected] >> Subject: Re: [core] CoMI remote server clash file and YANG hash in URL >> >> Hi, >> >> I do not agree that the module name or a module ID needs to be used at all >> in the object identifier. Each object has a YANG Hash. >> Retrieving a node, whether by hash ID or module-ID.hash will cause all >> descendant nodes to be included in the retrieval, even if they are >> augmenting nodes from another module. >> >> YANG data is hierarchical. >> The modules are just packaging for definitions. >> They do not define any containment or structure within the conceptual schema >> tree. >> >> >> Andy >> >> >> On Thu, Jun 4, 2015 at 8:21 AM, Michel Veillette >> <[email protected]> wrote: >>> Hi Andy >>> >>> About "Almost -- this is defined in RFC 6020 -- the augmenting nodes are >>> not part of the augmented module. They will be returned in NETCONF and >>> RESTCONF retrieval operations for an ancestor node but they are not part of >>> the augmented module. Only objects defined in the module namespace are >>> part of the module." >>> >>> Understood, this is why my contribution introduce the new term, "Module >>> context". Augmenting nodes are not part of the augmented module but, as >>> defined, are part of the "Module context". This new concept was require to >>> minimize the overhead when retrieving the complete datastore containing >>> multiple module contexts with some having augmenting nodes. >>> >>> RES: 2.05 Content (Content-Format: application/cbor) >>> { >>> "a" : { >>> 365257235 : "leafA1 value", >>> 702149626 : { >>> 215993329 : "leafA2 value", >>> 962191682 : "leafC1 value" # Augmenting data >>> node >>> }, >>> 829222983 : "leafS value" # Sub-module data >>> node >>> }, >>> "b" : { >>> 289564696 : "leafB value" >>> }, >>> "c" : { >>> 993533527 : "leafC2 value" >>> } >>> } >>> >>> Michel Veillette >>> System Architecture Director >>> Trilliant Inc. >>> Tel: 450-375-0556 ext. 237 >>> [email protected] >>> www.trilliantinc.com >>> >>> >>> -----Original Message----- >>> From: Andy Bierman [mailto:[email protected]] >>> Sent: 3 juin 2015 17:38 >>> To: Michel Veillette >>> Cc: Alexander Pelov; [email protected]; [email protected] >>> Subject: Re: [core] CoMI remote server clash file and YANG hash in URL >>> >>> On Wed, Jun 3, 2015 at 12:48 PM, Michel Veillette >>> <[email protected]> wrote: >>>> About "I would prefer if hash collisions in a single module were not >>>> allowed." >>>> Agreed but how this is implemented. >>>> >>> >>> >>> By checking the YANG hashes before it is published, similar to checking the >>> YANG syntax with a YANG compiler. >>> >>> >>>> About "But these numbers do not exist in any modules and so they cannot be >>>> relied on in CoMI. >>>> I understand but I assume that a module can be re-published with the added >>>> statement when require. If a CoMI implementer try to implement a module >>>> containing a clash, he can look for a version of this module containing >>>> the special YANG statement. The likelihood of such clash is extremely low >>>> and such fallback solution won’t be required often. However, the RFC will >>>> specify how to resolve such situation if this ever happen. This statement >>>> can also be used to address the need for data node identifier compactness >>>> as mentioned by Alexander in its original email. >>>> >>> >>> I think the ad-hoc id assignments are pointless because they are not >>> universal. The only thing that can be counted on is the module name and >>> the XPath absolute path expression of each object. >>> >>> >>>> About "In NETCONF, modules tend to use augment a lot" >>>> In my document " draft-vanderstok-core-comi-06 - MV.docx ", I propose a >>>> new definition to address this. >>>> >>> >>> What is it? >>> IMO you should post comments to the mailing list not markup to a word >>> document. It is too hard to follow. >>> >>> >>>> Module Context: A module context is composed of all data nodes, >>>> notifications and RPCs defined in a YANG module and included sub-modules >>>> or added to them using the augment YANG statement. >>>> >>> >>> Almost -- this is defined in RFC 6020 -- the augmenting nodes are not part >>> of the augmented module. They will be returned in NETCONF and RESTCONF >>> retrieval operations for an ancestor node but they are not part of the >>> augmented module. Only objects defined in the module namespace are part of >>> the module. >>> >>> >>> Andy >>> >>> >>> >>>> Section 4.1.3.4 show an example of include, import and augment. >>>> >>>> Michel Veillette >>>> System Architecture Director >>>> Trilliant Inc. >>>> Tel: 450-375-0556 ext. 237 >>>> [email protected] >>>> www.trilliantinc.com >>>> >>>> >>>> -----Original Message----- >>>> From: Andy Bierman [mailto:[email protected]] >>>> Sent: 3 juin 2015 14:54 >>>> To: Michel Veillette >>>> Cc: Alexander Pelov; [email protected]; [email protected] >>>> Subject: Re: [core] CoMI remote server clash file and YANG hash in >>>> URL >>>> >>>> On Wed, Jun 3, 2015 at 11:16 AM, Michel Veillette >>>> <[email protected]> wrote: >>>>> Hi Andy >>>>> >>>>> Couple of questions about your proposed solution. >>>>> >>>>> === Question #1, New YANG statement >>>>> >>>>> Your solution works only if hash clashes within a module are resolved at >>>>> design time. If not, the error message will return the following >>>>> information which is not sufficient to resolve which data node is which. >>>>> >>>>> { "1234": [ >>>>> "module-A":"4567", >>>>> "module-A":"6523" >>>>> ] >>>>> } >>>>> >>>> >>>> >>>> The original proposal had a 3-tuple { module-name, path-expr, new-hash } >>>> We discussed some optimizations in the 6TISCH call to reduce the size of >>>> the path-expr and still not require the client to store all the path >>>> strings, such as store a few bytes from the path strings like 6th and 11th >>>> bytes (Avoid the common ietf- prefix for many module names). >>>> >>>> This is not deterministic. Only the full path string is fool-proof. >>>> I would prefer if hash collisions in a single module were not allowed. >>>> They need to be resolved at design time. If this is not possible, then a >>>> standard rehash algorithm should be used so the collision is resolved in a >>>> predictable way. e.g., try appending "_", then try "__", etc. >>>> The result would be that hash collisions in a single module do not occur >>>> in CoMI. >>>> >>>> >>>>> Is it possible/acceptable to request a new YANG statement to manually >>>>> assign a ID to a data node? >>>>> Same question for a module ID? >>>>> >>>>> For example (see "id 16" and "id 25" bellow) : >>>>> >>>>> module a { >>>>> namespace a-ns; >>>>> prefix a; >>>>> id 16; >>>>> revision 2015-01-01; >>>>> >>>>> leaf leafA1 { type string; id 25 } >>>>> container containerA { >>>>> leaf leafA2 { type string} >>>>> } >>>>> } >>>> >>>> >>>> You could have an extension statement do anything you want: >>>> >>>> comi:id 16; >>>> >>>> But these numbers do not exist any any modules and so they cannot be >>>> relied on in CoMI. >>>> >>>> >>>>> >>>>> === Question #2, Introduction of an optional query parameter used to >>>>> target a specific module >>>>> >>>>> For nodes / networks which can afford a slitely larger payload (IEEE >>>>> 802.15.4g for example) but want to avoid the processing and storage >>>>> associated with the porposed error message, are you opposed to the >>>>> support of an optional CoAP query parameter use carry the module name or >>>>> module ID ? >>>>> >>>>> For example: >>>>> >>>>> GET example.com/Y8d7s?select= ietf-6tisch-mac >>>>> >>>>> === Question #3, Use of the select query parameter instead of the >>>>> URI >>>>> >>>>> Are you oposed to the use of the select query parameter instead of >>>>> the URI to identify data node(s) >>>>> >>>> >>>> Not really. >>>> >>>> I don't know how mixed the modules will get in CoMI. >>>> In NETCONF, modules tend to use augment a lot, so the nodes from >>>> 'module-A' may be nested nodes. >>>> Retrieving just the augmenting nodes and ignoring the augmented nodes >>>> would likely be operationally useless. >>>> e.g., when augmenting a list, the key leafs are needed to identify the >>>> instance. >>>> >>>> I would like to keep CoMI extremely simple. >>>> More bells and whistles means increased memory and CPU requirements. >>>> I am glad CORE and 6TISCH WGs are so concerned about constrained resources. >>>> Only the most important bells and whistles will get standardized. >>>> >>>> >>>>> For example: >>>>> >>>>> GET example.com/mg?select=Y8d7s >>>>> GET example.com/mg?select=Y8d7s,NKHaA GET >>>>> example.com/mg?select=ietf-6tisch-mac(Y8d7s,NKHaA) >>>>> >>>>> === Question #4, Compressing YANG hash base64 representation >>>>> >>>>> Are you oposed to the compression of the base64 representation as propsed >>>>> by Alexander. >>>>> >>>> >>>> No. Optimizations that reduce the encoding size are important. >>>> I want to use standard CBOR. No special hacks just for CoMI. >>>> >>>>> For example: >>>>> >>>>> GET example.com/mg?select=7s >>>>> Instead of >>>>> GET example.com/mg?select= AAA7s >>>>> >>>>> Michel Veillette >>>>> System Architecture Director >>>>> Trilliant Inc. >>>>> Tel: 450-375-0556 ext. 237 >>>>> [email protected] >>>>> www.trilliantinc.com >>>>> >>>>> >>>> >>>> Andy >>>> >>>>> -----Original Message----- >>>>> From: Andy Bierman [mailto:[email protected]] >>>>> Sent: 3 juin 2015 13:09 >>>>> To: Michel Veillette >>>>> Cc: Alexander Pelov; [email protected]; [email protected] >>>>> Subject: Re: [core] CoMI remote server clash file and YANG hash in >>>>> URL >>>>> >>>>> On Wed, Jun 3, 2015 at 9:05 AM, Michel Veillette >>>>> <[email protected]> wrote: >>>>>> Hi Alexander >>>>>> >>>>>> At the last 6TiSCH bi-weekly call, 4 solutions have been presented to >>>>>> the hash clashes problem. Two based on a rehash table and two based on a >>>>>> new CoAP query parameter carrying the module name or registered module >>>>>> ID. This query parameter is used to select unambiguously the targeted >>>>>> data node(s) assuming that any hash collision within a module have been >>>>>> resolved at design time. You can find this presentation in attachment. >>>>>> >>>>>> Both of your proposals make sense to me but I still hope that the clash >>>>>> file can be avoided or be optional. >>>>>> >>>>> >>>>> >>>>> I agree we should not need the client to retrieve the rehash info before >>>>> using a server. >>>>> The CoMI authors have discussed a solution which I will try to write up >>>>> today where a special "hash-clash" error is returned by CoMI if a rehash >>>>> is needed. >>>>> >>>>> 1) client requests module-A object with hash "1234"; >>>>> there is no extended-name/local-name, >>>>> just the YANG hash, used in any request >>>>> >>>>> 2) if '1234' has a collision in this server then a special error code >>>>> is returned; >>>>> The return payload has the rehash info [old-hash, { module-name, >>>>> new-hash }+] >>>>> { "1234": [ >>>>> "module-A":"4567", >>>>> "module-X":"6523" >>>>> ] >>>>> } >>>>> >>>>> 3) the client gets the error info, and knows which module its "1234" >>>>> is from. >>>>> It finds the correct module name in the array and replaces its >>>>> old-hash >>>>> with the new-hash. >>>>> >>>>> 4) client re-sends the request, using hash "4567" instead of "1234" >>>>> >>>>> The rehash table goes away, and only this error info is used instead. >>>>> >>>>>> Michel Veillette >>>>>> System Architecture Director >>>>>> Trilliant Inc. >>>>>> Tel: 450-375-0556 ext. 237 >>>>>> [email protected] >>>>>> www.trilliantinc.com >>>>>> >>>>> >>>>> >>>>> Andy >>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: core [mailto:[email protected]] On Behalf Of Alexander >>>>>> Pelov >>>>>> Sent: 3 juin 2015 11:01 >>>>>> To: [email protected]; [email protected] >>>>>> Subject: [core] CoMI remote server clash file and YANG hash in URL >>>>>> >>>>>> Hello everyone, >>>>>> >>>>>> I would like to discuss with you to ideas/proposals in this mail, namely: >>>>>> 1) Caching the yang clash file on a remote server >>>>>> 2) Compressing the URL representation of YANG hashes >>>>>> >>>>>> 1) Caching the yang clash file on a remote server Even though hash >>>>>> collisions should happen rarely (if ever), there may be cases when it >>>>>> could be inevitable, and as such, it may be interesting to cache the >>>>>> clash_file somewhere on a remote server to avoid sending it over the >>>>>> constrained wireless interface. It could be used for other cases, where >>>>>> some frequently used resources are re-hashed for performance purposes. >>>>>> Maybe it would be interesting to allow the server to specify a remote >>>>>> location for the clash file? For static collisions or optimizations it >>>>>> could be that the YANG definition includes alternative hash values, >>>>>> which could be used in addition to the normal ones. >>>>>> >>>>>> 2) Compressing the URL representation of YANG hashes Another point >>>>>> (which is unrelated), is to elide the leading "A"s from the base64 >>>>>> representation of the hash in the URL (which corresponds to eliding >>>>>> the leading zeroes from the hash). As specified in section 5.4. a >>>>>> hash value of 0x0000007e will be encoded as AAAA_, but if the >>>>>> leading A’s are elided it will be simply _ >>>>>> >>>>>> This could pose problems only if in the future the size of the hash >>>>>> values is increased from 30 bits to some other (higher) value. >>>>>> >>>>>> This, however, could be very useful when having to frequently >>>>>> access the same resource. In such case, the server/network operator >>>>>> may decide to re-hash the URI to a shorter version, e.g. the >>>>>> /sys:system-state/sys:clock/sys:current-datetime could be remapped >>>>>> from 0x15370408 and VNwQI to 0x1 and _ >>>>>> >>>>>> Best, >>>>>> Alex >>>>>> >>>>>> PS. >>>>>> A typo (?), which may be already corrected, in the example of mod.uri on >>>>>> Page 26: the two responses have different scheme "mod.uri" and "moduri" >>>>>> >>>>>> _______________________________________________ >>>>>> core mailing list >>>>>> [email protected] >>>>>> https://www.ietf.org/mailman/listinfo/core >>>>>> >>>>>> _______________________________________________ >>>>>> core mailing list >>>>>> [email protected] >>>>>> https://www.ietf.org/mailman/listinfo/core >>>>>> > _______________________________________________ 6tisch mailing list [email protected] https://www.ietf.org/mailman/listinfo/6tisch
