===== About this concept of "handle table generation off-line". I don’t understand how this approach work, this seem a chicken and egg issue
The table need to have: - A identifier of a peer CoMI server (IPv6 address) - The rehash value use on this server (Unsigned 32 bits) - The identifier of the attribute affected (If not the canonical representation of the path identifier, what it is?) Considering also the number of peer CoMI servers in some application space (thousands of peers in a NAN network for example), I don't see this as a viable solution. ===== About "we can make the module identifier optional, and specify it in a query parameter" How can we make the module identifier optional and avoid interoperability issues? If we accept the introduction of a managed module identifier, I recommend to make it mandatory. ===== About " avoids transporting the module identifier when it is not needed" For me, the current discussion is about managed vs. unmanaged identifiers, not about protocol overhead. If we want to discuss overhead, we should create an independent thread on this subject to discuss things like support of managed data node identifiers which can reduce significantly the payload size even with the presence of the module identifier. Michel Veillette System Architecture Director Trilliant Inc. Tel: 450-375-0556 ext. 237 [email protected] www.trilliantinc.com -----Original Message----- From: peter van der Stok [mailto:[email protected]] Sent: 15 avril 2015 03:00 To: Michel Veillette Cc: [email protected]; Andy Bierman; Pascal Thubert (pthubert); Carsten Bormann; [email protected]; Core Subject: RE: [6tisch] [core] COMI hash values globally unique vs. unique within a module HI Michel, Your scenario perfectly fits in with my summary. The proposal is that each server is accompanied by a file that mentions the names that are rehashed for a given device. Of course, you can generate unique hash values for a whole installation for which you need an installation repository, but as pointed out by Andy, new servers with new modules are added and new modules are added to existing servers, which invalidates the installation repository. Assuming that "small" clients do not dynamically discover servers, and that the code (or access tables) are generated offline and down-loaded into the clients, you do not need an installation repository but can handle the table generation for a client with only the clash information of the involved servers. The disadvantage is that you need a table in the client that gives the hash value as function of the server identity. That table does not need to be large, given the very low clash probability (and may be non-existent for most clients). However, when clients try to build these tables dynamically, and the clients do not have space to store all accessible names, and they dynamically discover the servers, then you need the module identifiers, which need a centrally managed repository. I thought that for 6tisch all servers were kind of fixed with no changes to the module contents; and clients were small accessing a non changing set of servers. In that case I would suggest to handle table generation off-line. In the clients the hash values can be the same for all servers, with 0-3 exceptions where a given server uses a different hash. For the dynamic case, with small clients that cannot store the names, I am afraid that using module identifiers is a better solution. However, we can make the module identifier optional, and specify it in a query parameter. That avoids transporting the module identifier when it is not needed. Peter Michel Veillette schreef op 2015-04-14 18:59: > Hi Peter > > I’m not sure if your summary capture the impossibility to avoid > multiple set of YANG hashes for a same module during a deployment > > unless a central repository of hashes is established from the start > for all standard and manufacture modules. > > For example > > · Let assume the following distribution of YANG hashes > > · Let also assume that modules have no hash clashes or data node > identifier re-assignment have been done offline. > > · Let assume that device x implements module C > > · Then, device y implements module B and module C, hash clashes A/C > need to be resolved to create Module C’ > > · Finally, device z is introduced with support of modules A, B and C, > both hash clashes A/C and B/C need to be resolved to create Module C’’ > > At the end of this sequence, we have three devices (x, y, and z) which > all implement module C with three different version of YANG hashes. To > make this work, the proposed rehash mechanism is needed and each node > need to dynamically discover rehash table from each peer and need tens > of kB of URI strings for the lockup in these rehash tables. > > Unless we want to put in place this central repository of YANG hashes, > scoping the uniqueness of data node identifiers by module seem to be > the only solution. > > Michel Veillette > > System Architecture Director > > Trilliant Inc. > > Tel: 450-375-0556 ext. 237 > > [email protected] > > www.trilliantinc.com > > -----Original Message----- > From: peter van der Stok [mailto:[email protected]] > Sent: 14 avril 2015 05:06 > To: Andy Bierman > Cc: [email protected]; Michel Veillette; Pascal Thubert > (pthubert); Carsten Bormann; [email protected]; Core > Subject: Re: [6tisch] [core] COMI hash values globally unique vs. > unique within a module > > Hi Andy, > > From Michel and you I understand that the problems with rehashing come > > > from: > > - Different manufacturers may provide servers with the same module > set, but may have chosen different names to rehash in case of clash. > > - Servers may be upgraded with modules, and may need rehashing. > > Not choosing module identifiers in the name hash, has as consequence > that in case of clash in a server, the server needs to announce the > name that is rehashed. > > This announcement can be done in a standardized format and pre-loaded > servers with clashes MUST be accompanied by such a file. > > Given that small clients need to be prepared when servers are added or > modules are added to existing servers, Appropriate code can be loaded > in these clients when needed. > > The existence of rehashes does not unnecessarily complicate this work. > > > The choice is then between: > > - no module IDs: rehashes need to be administrated and taken into > account when servers are added or functionality is added. > > - module ID: Registry needs to be defined, maintained, and the module > ID must be transported in every exchange. > > The module ID is compulsory when the client does an automatic > discovery of servers and needs to use the same hash values independent > of the arrival of new servers or the extension of servers. > > Is this a correct summary of the arguments involved? > > Peter > > Andy Bierman schreef op 2015-04-13 18:14: > >> On Mon, Apr 13, 2015 at 1:34 AM, peter van der Stok > >> <[email protected]> wrote: > >>> Dear all, > >>> > >>> Trying to work out all your recommendations, I fail to understand > the > >>> need for the solution. > >>> > >>> Large clients and small servers is not tru any more. > >>> The assumption is that clients and servers are small, they are not > >>> dynamically augmented with new modules. They only know about hash > >>> values and have no knowledge what so ever of the corresponding > names. > >>> > >> > >> The term "dynamically loaded" is misleading, because a new revision > of > >> the device, or a device that supports firmware upgrade, can have new > > >> YANG modules added. > >> > >> > >>> Let's concentrate on the one module clash first. > >>> Apparently, it is envisaged to load modules that already contain > hash > >>> clashes inside. > >>> For me that is an unsolvable problem: There is one hash value that > >>> points to different names (memory stores) in the server. > >>> Servers only know about hash values, they have no name tables. > >>> Suppose one of the names has been rehashed (where? certainly not > >>> within the server). > >>> We might add a statement in the server that the old hash value goes > > >>> to the new rehashed value. > >>> It is not clear to me how the server can decide which memory > location > >>> goes with the old hash value and which with the new value. > >>> > >> > >> If a value gets rehashed, it is because there is a collision. > >> I am not sure if module numbers can be added to YANG, but it should > be > >> possible to either avoid 1-module collisions or add a YANG extension > > >> to the module to resolve the collision so that every implementation > >> uses the same rehash (known in advance). > >> > >> > >>> Apparently, something needs to be done before the module is loaded > >>> into the server. For me that means that only modules can be loaded > >>> without hash clashes. > >>> > >> > >> For modules intended for CoMI, we can certainly make sure no names > >> used in the module produce any collisions. > >> > >> > >>> Suppose in a given server two names from two modules clash. The > >>> solution is then that the module ID distinguishes between the two > >>> clashing values. > >>> It is then proposed that module names are hashed. That solution is > >>> the same as saying that you need more than 32 bits for a hash value > > >>> to reduce clash probability. > >>> > >>> The only valid remaining approach is a registry that maps module > name > >>> to a > >>> 32 bit? ID. Transporting the module ID means additional transport > >>> costs. > >>> > >> > >> It should be possible to come up with a short module-id format, and > >> only use the long form (full module name) if no short-form exists. > >> > >>> I can imagine though that in a given installation the servers and > >>> their module combinations are known. > >>> Any clashes can be solved before load time. The appropriate code > can > >>> be loaded in the small clients and servers. > >>> > >> > >> This does not allow for the old client/new server scenario to keep > >> working, if the new server adds a module with a hash collision. > >> > >>> The problem arrives when a new server with a set of modules > >>> (including a new > >>> module) finds a clash. > >>> This clash does not affect the old clients and servers (they are > not > >>> aware of the new module) > >> > >> But what if the module they know about is the one that gets > rehashed? > >> There is no saved or canonical order for processing YANG modules. > >> > >>> Consequently, the clash can be solved by renaming the clashing name > > >>> in the newly added module. > >>> > >> > >> Only if the collision is detected and corrected before > >> the new module is published. > >> > >> > >>> All clients, and servers which have to use the clashing name of the > > >>> new > >>> module will use the hash value of the renamed name, and do not need > to > >>> be > >>> aware of the rehashing. > >>> > >>> At the operational level, there are only unique hashes within an > >>> installation. > >>> At the organization level, the rehash value of the new module needs > to > >>> be > >>> registered and used when code for new clients or servers are > >>> generated. > >>> > >>> Consequently, I do not see a need for module IDs. > >> > >> > >> YANG is modular, and different naming authorities (SDOs and vendors) > > >> work at different speeds and publish independently of each other. > >> Applications can use different modules on the same server without > >> impacting each other. Vendors can add their own modules before > >> or after standard modules are written and added. > >> > >> If module A and module B both have objects that hash to the same > value > >> then they cannot be used together without re-hashing 1 of the > objects. > >> > >>> > >>> Any mistakes in the above? > >> > >> I think the combinations of modules that will be available > >> on a given implementation cannot be controlled in advance. > >> We cannot assume all client devices and server devices can > >> be upgraded together (flag day upgrade). > >> > >> > >>> > >>> Peter > >>> > >> > >> > >> Andy > >> > >>> Michel Veillette schreef op 2015-04-02 20:37: > >>>> > >>>> If I summarize: > >>>> > >>>> Reducing the scope of uniqueness of YAND hash values to each > module > >>>> allows detection of hash collision(s) at design time instead of at > > >>>> run > >>>> time. > >>>> > >>>> To reduce this scope, we need to: > >>>> - Create a unique module identifier (e.g. 20 bits organization ID > >>>> registered at IANA, 10 bits module ID) > >>>> - Add this module identifier to the URI (e.g. one to five base64 > >>>> characters) > >>>> - Add a map at the root of the CBOR objects carry within the CoAP > >>>> payload to associate module IDs with modules data nodes (e.g. one > to > >>>> five byes per module) > >>>> > >>>> The problem that still need to be resolved is how we document data > > >>>> nodes rehash offline. > >>>> - One option is to use the description statement of the YANG > module > >>>> or > >>>> a new YANG statement specifically added for this purpose. > >>>> - Another option is create a IANA registry for these rehash > values. > >>>> > >>>> Hash collisions within YANG modules are very unlikely to happen > and > >>>> the solution don't need to be highly scalable. In that respect, > using > >>>> a IANA registry might be a good solution since it can be used for > >>>> already published yang modules and can be use after the fact with > any > >>>> future YANG modules. > >>>> > >>>> Michel Veillette > >>>> System Architecture Director > >>>> Trilliant Inc. > >>>> Tel: 450-375-0556 ext. 237 > >>>> [email protected] > >>>> www.trilliantinc.com [1] > >>>> > >>>> > >>>> -----Original Message----- > >>>> From: Pascal Thubert (pthubert) [mailto:[email protected]] > >>>> Sent: 2 avril 2015 13:09 > >>>> To: Carsten Bormann; Andy Bierman > >>>> Cc: Michel Veillette; [email protected]; [email protected]; > > >>>> [email protected] > >>>> Subject: RE: [6tisch] [core] COMI hash values globally unique vs. > >>>> unique within a module > >>>> > >>>> Yes, we asked about that as well. > >>>> > >>>> It help to have a registry that guarantees the uniqueness of the > >>>> first > >>>> element in the path, so as to make it easier to check for > collision > >>>> within one path only. > >>>> > >>>> Cheers, > >>>> > >>>> Pascal > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: 6tisch [mailto:[email protected]] On Behalf Of > Carsten > >>>>> Bormann > >>>>> Sent: jeudi 2 avril 2015 18:41 > >>>>> To: Andy Bierman > >>>>> Cc: Michel Veillette; [email protected]; > [email protected]; > >>>>> [email protected] > >>>>> Subject: Re: [6tisch] [core] COMI hash values globally unique vs. > > >>>>> unique within a module > >>>>> > >>>>> I think I'm losing track of what "this" is. > >>>>> Assigning numbers when doing a (version of a) module should be > >>>>> doable, > >>>>> even if it requires some additional tooling. > >>>>> Assigning numbers to modules would require a registry. > >>>>> No rehashing (or hashing at all) required. > >>>>> (Now, that registry could be filled using a hash...) > >>>>> > >>>>> BTW, I'm not sure that the "rehashing" in the current spec is the > > >>>>> easiest way to handle collisions -- I also don't understand how > >>>>> multiple paths that arrive at the same hash value are handled. > >>>>> > >>>>> Grüße, Carsten > >>>>> > >>>>> Andy Bierman wrote: > >>>>> > On Thu, Apr 2, 2015 at 8:25 AM, Andy Bierman > <[email protected]> > >>>>> wrote: > >>>>> >> Hi, > >>>>> >> > >>>>> >> I should make it clear that I was trying to use a module-id so > the > >>>>> >> objects in the module could be given a simple integer OID. > >>>>> >> The goal was to support identifiers that could fit in 1 or 2 > bytes. > >>>>> >> But this was dropped when we changed to using hashes. > >>>>> >> > >>>>> >> It might be a huge effort to create and use numeric module > IDs, but > >>>>> >> YANG module names tend to be long strings. It might be worth > it, if > >>>>> >> a per-module hash is used. > >>>>> >> > >>>>> >> If the collisions were coupled to specific modules, then they > might > >>>>> >> be avoided before publication or the client can be hard-wired > to > >>>>> >> use the rehashed values from the start. > >>>>> >> > >>>>> >> A client would not need to know the XPath strings for any > objects. > >>>>> >> It could be hard-coded to work with module X, Y, Z, and it > would > >>>>> >> always work, no matter what additional modules were added to > the > >>>>> >> server. > >>>>> >> This does seem very useful for constrained clients. > >>>>> >> > >>>>> > > >>>>> > A standard, deterministic renaming algorithm would be needed to > make > >>>>> > this viable. If each server is free to choose its own rehash, > then > >>>>> > the client still needs to store all the strings, and resolve > the > >>>>> > collisions at run-time instead of compile-time. > >>>>> > (So therefore a module-name or module-id would not solve > anything). > >>>>> > > >>>>> >> Andy > >>>>> > > >>>>> > Andy > >>>>> > > >>>>> >> > >>>>> >> On Thu, Apr 2, 2015 at 5:55 AM, Carsten Bormann <[email protected]> > wrote: > >>>>> >>> Michel Veillette wrote: > >>>>> >>>> strings represents */5323 bytes/* > >>>>> >>> If we want to free the implementations completely of having > to use > >>>>> >>> these strings, we'll need to assign numbers. This can be done > at > >>>>> >>> the module level, either manually or algorithmically. Using a > > >>>>> >>> per-module hash to do this is suboptimal; it is much better > to > >>>>> >>> fill a small > >>>>> linear space. > >>>>> >>> > >>>>> >>> We then need a way to compose these numbers with numbers > assigned > >>>>> >>> to module IDs. SMIv2 had this way to name modules and their > >>>>> >>> components... > >>>>> >>> > >>>>> >>> Grüße, Carsten > >>>>> > > >>>>> > _______________________________________________ > >>>>> > 6tisch mailing list > >>>>> > [email protected] > >>>>> > https://www.ietf.org/mailman/listinfo/6tisch [2] > >>>>> > > >>>>> > > >>>>> > >>>>> _______________________________________________ > >>>>> 6tisch mailing list > >>>>> [email protected] > >>>>> https://www.ietf.org/mailman/listinfo/6tisch [2] > > Links: > ------ > [1] http://www.trilliantinc.com > [2] https://www.ietf.org/mailman/listinfo/6tisch _______________________________________________ 6tisch mailing list [email protected] https://www.ietf.org/mailman/listinfo/6tisch
