On Thu, 2016-12-01 at 14:25 +0100, Josef Skladanka wrote:
> On Wed, Nov 30, 2016 at 6:29 PM, Adam Williamson <adamw...@fedoraproject.org
> > wrote:
> > On Wed, 2016-11-30 at 18:20 +0100, Josef Skladanka wrote:
> > > I would try not to go the third way, because that is really prone to
> > 
> > erros
> > > IMO, and I'm not sure that "per context" is always right. So for me, the
> > > "TCMS" part of the data, should be:
> > > 1) testcases (with required fields/types of the fields in the "result
> > > response"
> > > 2) testplans - which testcases, possibly organized into groups. Maybe
> > 
> > even
> > > dependencies + saying "I need testcase X to pass, Y can be pass or warn,
> > 
> > Z
> > > can be whatever when A passes, for the testplan to pass"
> > > 
> > > But this is fairly complex thing, to be honest, and it would be the first
> > > and only useable TCMS in the world (from my point of view).
> > 
> > I have rather different opinions, actually...but I'm not working on
> > this right now and I'd rather have something concrete to discuss than
> > just opinions :)
> > 
> > We should obviously set goals properly, before diving into implementation
> details :) I'm interested in what you have in mind, since I've been
> thinking about this particular kind of thing for the last few years, and it
> really depends on what you expect of the system.

Well, the biggest point where I differ is that I think your 'third way'
is kind of unavoidable. For all kinds of reasons.

We re-use test cases between package update testing, Test Days, and
release validation testing, for instance; some tests are more or less
unique to some specific process, but certainly not all of them. The
desired test environments may be significantly different in these
different cases.

We also have secondary arch teams using release validation processes
similar to the primary arch process: they use many of the same test
cases, but the desired test environments are of course not the same.

Of course, in a non-wiki based system you could plausibly argue that a
test case could be stored along with *all* of its possible
environments, and then the configuration for a specific test event
could include the information as to which environments are relevant
and/or required for that test event. But at that point I think you're
rather splitting hairs...

In my original vision of 'relval NG' the test environment wouldn't
actually exist at all, BTW. I was hoping we could simply list test
cases, and the user could choose the image they were testing, and the
image would serve as the 'test environment'. But on second thought
that's unsustainable as there are things like BIOS vs. UEFI where we
may want to run the same test on the same image and consider it a
different result. The only way we could stick to my original vision
there would be to present 'same test, different environment' as another
row in the UI, kinda like we do for 'two-dimensional test tables' in
Wikitcms; it's not actually horrible UI, but I don't think we'd want to
pretend in the backend that these were two completely different. I
mean, we could. Ultimately a 'test case' is going to be a database row
with a URL and a numeric ID. We don't *have* to say the URL key is
unique. ;)

There are really all kinds of ways you can structure it, but I think
fundamentally they'd all boil down to the same inherent level of
complexity; some of them might be demonstrably worse than others
(like...sticking them all in wikicode and parsing wiki table syntax to
figure out when you have different 'test instances' for the same test
case! that sounds like a *really bad* way to do it!)

Er. I'm rambling, aren't I? One reason I actually tend to prefer just
sitting down and writing something to trying to plan it all out
comprehensively is that when I just sit here and try to think out
planning questions I get very long-winded and fuzzy and chase off down
all possible paths. Just writing a damn thing is usually quite quick
and crystallizes a lot of the questions wonderfully...
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Reply via email to