It is true that we don’t want the information about “schema” of metadata record to be scattered all over, but there is a single place where this information is always correct are describes the current format of the records - tupleTranslators, which serialize\deserializes metadata records into/from objects. Right now they look like bunch of boilerplate code, however if they were to be refactored with handy util methods that would have been the best “documentation” for the current state of the world, no?
> On Dec 14, 2015, at 13:19, Mike Carey <[email protected]> wrote: > > I think we started out as a bunch of relation-heads, modeling things after > the system catalogs found in our favorite RDBMSs. Their record types are > open AFAIK - but I think Steven was saying that it feels like this is a use > case whose field(s) should be predeclared (and using the term "closed field" > to mean predeclared field). I agree. I think to the extent that we can > anticipate the need for a given bit of metadata, it's useful to have it be > there in the known schema; but, when we move into the era when we cannot make > changes to the known schema part of the catalogs, because we have dependent > customers who'd be hurt by that, we will start adding more fields that are > not predeclared but only documented. The schema for the metadata is really > there more as documentation than as being "truly necessary"... > > When we first started out we were just not thinking in an "open-minded" way > on the catalog side of things..... :-) > > Cheers, > Mike > > On 12/14/15 11:53 AM, Till Westmann wrote: >> Let me prefix this with that statement that I haven’t looked into our >> metadata storage implementation. >> But looking at the relatively frequent changes to the metadata format it >> seems that we should either >> a) make the types for metadata records open or >> b) be very careful in designing the next (and future) revision(s) of it. >> I don’t know why the metadata storage was done the way it is now and what >> the considerations were when we did that. >> >> Does anybody remember those (or is there even a document that contains the >> design and rationale)? >> >> Thanks, >> Till >> >> On 14 Dec 2015, at 11:35, Steven Jacobs wrote: >> >>> It could easily be done as reverse-compatible, but my thinking was that >>> this is the "wrong" choice. I can easily make the datatype dataverse an >>> open field for the metadata. The question is, why do we have open vs closed >>> fields for metadata at all? If it is okay for them to be open, should we >>> get rid of the schema entirely? if it's not okay, then shouldn't this field >>> be closed? From a design standpoint it seems that if reverse compatibility >>> were not an issue than the field should be closed. >>> Steven >>> >>> On Mon, Dec 14, 2015 at 11:30 AM, Ildar Absalyamov < >>> [email protected]> wrote: >>> >>>> If fix for 2) will break the backwards-compatibility 1) might do that as >>>> well and be submitted together. >>>> >>>> Now 2) was a long overdue problem, I don’t think there is any reason even >>>> to try make changes backwards-compatible, because it was broken in the >>>> first place. >>>> >>>>> On Dec 14, 2015, at 11:16, Steven Jacobs <[email protected]> wrote: >>>>> >>>>> It's a new attribute, but it's a closed field, which means it isn't >>>>> backwards compatible. >>>>> Steven >>>>> >>>>> On Mon, Dec 14, 2015 at 11:13 AM, Ian Maxon <[email protected]> wrote: >>>>> >>>>>> For 1), I guess the question is whether it would be a backwards >>>>>> compatible change. Since it's just a new attribute (right?...), and it >>>>>> is also sort of a new feature rather than a fix for something that was >>>>>> critically broken, I would tend toward putting it on master. If it's >>>>>> not backwards compatible though maybe it needs more careful >>>>>> consideration. >>>>>> >>>>>> -Ian >>>>>> >>>>>> On Mon, Dec 14, 2015 at 10:12 AM, Steven Jacobs <[email protected]> >>>> wrote: >>>>>>> Hi all, >>>>>>> I'm implementing a change so that datasets can use datatypes from >>>>>> alternate >>>>>>> data verses (previously the type and set had to be from the same >>>>>>> dataverse). Unfortunately this means another change for Dataset >>>> Metadata >>>>>>> (which will now store the dataverse for its type). >>>>>>> >>>>>>> As such, I had a couple of questions: >>>>>>> >>>>>>> 1) Should this change be thrown into the release branch, as it is >>>> another >>>>>>> Metadata change? >>>>>>> >>>>>>> 2) In implementing this change, I've been looking at the Metadata >>>>>> secondary >>>>>>> indexes. I had a discussion with Ildar, and it seems the thread on >>>>>> Metadata >>>>>>> secondary indexes being "hacked" has been lost. Is this also something >>>>>> that >>>>>>> should get into the release? Is there anyone currently looking at it? >>>>>>> >>>>>>> Steven >>>>>> >>>> >>>> Best regards, >>>> Ildar >>>> >>>> > Best regards, Ildar
