On 30/04/2012 19:15, Roman Grigoryev wrote: > Hi Cris, > I'm glad to read next emails on this direction. > Please don't consider this as criticism, I just would like to get more > clearness: what is target of adding this metadata? Do you have plans to > use the metadata in other scripts? How? Does this metadata go to to results? > > Also please see more my comments inline: The metadata can be used in a multitude of ways, for example we can create dynamic test sets based on the changes made or target area of testing. What we are doing here is creating an understanding of the tests that we have so that we can improve our processes and testing capabilities in the future.
The metadata does not go to the results. The metadata is a database in it's own right and should metadata about a test be required it would be accessed from the source (database) itself. > On 04/30/2012 08:50 PM, Chris wrote: ... snip ... >> Prerequisites: Pre-requisite tests that must be run before this test >> can be run. This is again an array which presumes a test may have >> multiple pre-requisites, but the data should not contain a chain of >> prerequisites, i.e. if A requires B and B requires C, the pre-requisites >> of A is B not B& C. > On which step do you want to check chains? And what is logical base for > this prerequisites exclude case that current tests have hidden > dependencies? > I don't see any difference between one test which have body from tests > a,b,c and this prerequisites definition. > Could you please explain more why we need this field? As I said we can mine this data any-time and anyway that we want, and the purpose of this discussion is the data not how we use it. But as an example something that dynamically built test sets would need to know prerequisites. The suffix of a,b,c could be used to generate prerequisite information but it is firstly inflexible, for example I bet 'b','c' and 'd' are often dependent on 'a' but not each other, secondly and more importantly we want a standard form for storing metadata because we want to introduce order and knowledge into the test scripts that we have today. >> TicketIDs: This is an array of ticket numbers that this test >> explicitly tests. In theory we should aim for the state where every >> ticket has a test associated with it, and in future we should be able to >> carry out a gap analysis. >> > I suggest add keywords(Components could be translated as keywords too) > and test type (stress, benchmark, load, functional, negative, etc) for > quick filtering. For example, SLOW could transform to keyword. This seems like a reasonable idea although we need a name that describes what it is, we will need to define that set of possible words as we need to with the Components elements. What should this field be called - we should not reduce the value of this data why genericizing it into 'keywords'. > Also, I would like to mention, we have 3 different logical types of data: > 1) just human-readable descriptions > 2) filtering and targeting fields (Componens, keywords if you agree with > my suggestion) > 3) framework directives(Prerequisites) > >> As time goes on we may well expand this compulsory list, but this is I >> believe a sensible starting place. >> >> Being part of the source this data will be subject to the same review >> process as any other change and so we cannot store dynamic data here, >> such as pass rates etc. > What you you think, maybe it is good idea to keep metadata separately? > This can be useful for simplifying changing data via script for mass > modification also as adding tickets and pass rate and execution time on > 'gold' configurations? It would be easier to store the data separately and we could use Maloo but it's very important that this data becomes part of the Lustre 'source' so that everybody can benefit from it. Adding tickets is not a problem as part of the resolution issue is to ensure that at least one test exercises the problem and proves it has been fixed, the fact that this assurance process requires active interaction by an engineer with the scripts is a positive. As for pass rate, execution time and gold configurations this information is just not 1 dimensional enough to store in the source. Chris _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
