With the ARM moving to primary and essentially cloud achieving the same
status (from a QA perspectice), we need a better way to track and
display test results for each release. The current method (1) is a wiki
page with a set of columns for each architecture and rows for test cases
grouped by test case type. While this is easily expandable, it quickly
becomes a less than optimal method to display and manage data.
What Adam and I were discussing is a web app to do this. The problem is
that neither of us are UI designers. We have some ideas as to how we
want information displayed, queried, etc, but that's it. Here are the
relevant irc logs.
16:08 < handsome_pirate> adamw: Also, it's probably too late for F20, but I've
been thinking about the huge number of columns in the test matrices issue
16:08 < handsome_pirate> adamw: Namely, I've been kicking around doing a web
app
16:09 <@dgilmore> adamw: generally pxe with kickstart, but pxe and interactive
works also
16:09 < adamw> handsome_pirate: yeah, too late for f20, but i'd love a more
capable replacement
16:09 < adamw> handsome_pirate: i have a plan for f20 that's much simpler:
table the other way around
16:09 < handsome_pirate> Indeed
16:10 < adamw> so for this test case, it'll be a table on the install matrix
page but with the *test case* as the columns and the platforms as the rows
16:10 < handsome_pirate> adamw: I'll see if I can get my programming
instructor to let me do it for class credit
16:10 < adamw> probably have multiple columns representing the same test case
with different images
16:10 < adamw> that was my next thing to do
16:10 < adamw> handsome_pirate: it might turn into kind of a big project, note
16:11 < handsome_pirate> adamw: Indeed
16:11 < adamw> handsome_pirate: if we're going to do something better then we'd
want to do something awesome, that hooks into the blocker bug webapp etc
16:11 < adamw> but don't let me project stop energy at you :P
16:11 < handsome_pirate> adamw: But, I'm not a GSoC person; I won't be going
anywhere
16:11 < handsome_pirate> adamw: That would be wicked cool
16:11 < handsome_pirate> adamw: But, for starters, I want to follow KISS
16:12 < handsome_pirate> adamw: I'll ping mizmo or emichan to see about making
it pretty
16:13 * handsome_pirate noms on steak and bacon tacos
16:15 < adamw> handsome_pirate: yeah, in fact, thinking about it, the matrices
are the most easily drop-in-replacable thing
16:15 < adamw> you could just do a webapp which represents that layout more
flexibly and that would drop right in and make things better
16:15 < adamw> good thinking
16:16 < handsome_pirate> Exactly what I was thinking to start out
16:17 < adamw> cool
16:17 < adamw> let's see, if i wrote a five minute spec for you...
16:17 < handsome_pirate> Aye
16:17 < adamw> we'd need a concept of tests, configurations, and releases
16:17 < adamw> and then a nice interface for querying the database based on
those concepts
16:18 -!- streeter [streeter@nat/redhat/x-asubidnuwbirqiyg] has quit [Quit:
Leaving]
16:18 < adamw> oh, and test level of course...and that might need to vary
depending on the other concepts...hm
16:19 < adamw> handsome_pirate: so we need to express stuff like 'this test
needs to be run on this config at alpha, on this config at beta, can optionally
be run on this config, and is not applicable to this other one'
16:19 < adamw> and then some sort of awesome query engine so we can generate
all sorts of different views, for working from and for searching
16:19 < adamw> SOUNDS EASY RITE?!
16:19 < handsome_pirate> adamw: For db query, maybe have some way to select a
test case and then it shows a page with current results for that test case for
all images
16:19 < handsome_pirate> heh
16:20 < adamw> oh and we need test category too
16:20 < handsome_pirate> Indeed
16:20 -!- tempus_fol [~tempus@gateway/tor-sasl/tempusfol] has joined #fedora-qa
16:20 < Cerlyn> Just don't try and do it within Mediawiki's own internal db
unless they've made it so you don't have to flush every page to see an update
16:20 < adamw> sure, but i'm thinking stuff like 'show me all the required
deployments tests for ARM that we haven't done since RC1'
16:20 < handsome_pirate> I've got something sort of floating around in my head
16:20 < adamw> Cerlyn: handsome's thinking of writing a webapp for it
16:20 < handsome_pirate> adamw: Aye
16:21 < Cerlyn> Yet another test case manager :/
16:21 < handsome_pirate> Cerlyn: Probably use something like sqllite
16:21 < handsome_pirate> Cerlyn: Do you think there's one out there we can
adopt?
16:21 -!- weld [[email protected]] has joined #fedora-qa
16:22 < Cerlyn> handsome_pirate: I'd have to look. At one point Fedora was
going to move to what RH was moving to (Nitrate) but that seem to have stopped
16:22 < handsome_pirate> adamw: Honestly, the hardest part for me is going to
be the UI
16:22 < adamw> handsome_pirate: we're growing more and more test cases and test
case sets and platforms, the wiki approach is definitely getting unwieldy
16:22 < Cerlyn> Mozilla was working on one which was supposed to be OSS &
public friendly
16:22 < handsome_pirate> adamw: Indeed
16:22 < adamw> handsome_pirate: actually the more i think about it the more
this would be awesome and we need it NOW
16:22 < handsome_pirate> adamw: Okay, I'll set to work
16:23 < adamw> Cerlyn: the neat thing here is we're just looking at a UI for
the *matrices*
16:23 < adamw> Cerlyn: we're doing an end run around the whole tcms problem
16:23 < adamw> Cerlyn: the test cases can still be dumb wiki pages; what we
really could do with is a much more flexible way of querying test case sets
(and entering results)
16:24 < handsome_pirate> Cerlyn: All the test case managers I've seen won't do
what we need to do
16:24 < Cerlyn> So your TCMS will import the Wiki data as a source
16:24 < adamw> Cerlyn: it doesn't need any data as a source. it's not a tcms.
16:24 < adamw> it's a webapp for representing views on the set of test cases.
16:24 < adamw> all it needs to know is the URLs of the test case pages. it does
not care a jot about the contents.
16:25 < Cerlyn> then how does it know which test cases haven't been run yet?
16:25 < adamw> Cerlyn: we could build a simple one which just lets us look at
groups of test cases and even that would be interesting, but yeah, we need it
to store that
16:26 < adamw> handsome_pirate: i've been using trac's query interface a lot
the last few days and it has its good points. i'm kinda thinking of something
like that - a nice quick GUI 'query builder' which lets you generate all kinds
of different views and save useful ones. we'd probably have a UI with some kind
of overview for the current test build as the default, a set of canned queries,
and the ui for building a custom one
16:27 -!- tyll [~till@fedora/tyll] has quit [Ping timeout: 246 seconds]
16:27 < adamw> Cerlyn: it shouldn't be _too_ hard to store that info, though.
it really just needs to store...let's see, user foo ran test bar on
configuration moo for release baa on (date)
16:28 < adamw> with result X, associated bug reports Y, freeform comment Z
16:28 < adamw> did I miss anything?
16:28 < handsome_pirate> adamw: I'm thinking that for the main overview (ie,
where you could input results, get a quick view, etc), have a list of current
test cases
16:29 < handsome_pirate> Click on one, and it expands to show current results
and a place to input results
16:29 < handsome_pirate> Does that make sense?
16:29 < handsome_pirate> I'll follow trac's query page for setting up my own
16:29 < adamw> you might be best asking design team, like you said :) but let
me think
16:29 < adamw> if we put ALL the test cases in one big list that's gonna be a
bit overwhelming...
16:30 < handsome_pirate> True
16:30 * adamw tries to think what a UI would look like from a completely white
page and is utterly terrified
16:30 * adamw never wants to be a designer
16:31 < handsome_pirate> I think the click on test case, it expands with info
(at the least, link to wiki), test results, and place to enter test results may
be a starting point
16:31 < handsome_pirate> I'll write up an email for the design list
16:31 < adamw> sure, if you just want to have a placeholder UI while you work
on it it sounds fine
16:31 -!- Cerlyn [[email protected]] has quit
[Remote host closed the connection]
16:32 < adamw> i suppose if you query the list in a certain way and then hit
'enter result' it should pre-fill the result thingy with your query restrictions
16:32 < adamw> (so if you query for ARM tests for F20 Alpha RC2, it should give
you a report form with that config and release selected...)
16:32 < adamw> zoiks, i need to get back to working on what we have now, hehe
:) i'll leave you to it
At this point, any help with figuring out how to make it look would be
much appreciated.
Thanks much,
John.
(1): Example:
https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC1_Install
_______________________________________________
design-team mailing list
[email protected]
https://lists.fedoraproject.org/mailman/listinfo/design-team
