On Sat, 2014-10-11 at 02:37 -0700, Adam Williamson wrote:
> Hey, folks. Today I got a PoC of testcase-stats on top of wikitcms up
> and running. It's not complete yet as I haven't written the bits to
> provide the CLI and page selection stuff, but all the hard stuff is
> there. I'm attaching it. To test it, just stick it in the
> stats/testcase-stats directory of a fedora-qa git checkout, symlink an
> up-to-date checkout of the wikitcms 'results-types' branch into the same
> directory, and run it. It'll output to /tmp/tcstats_test , but you can
> easily change that if you like. You can also change the hard-coded list
> of pages it runs on.
>
> Tomorrow I'll aim to clean it all up and make it a relval sub-command,
> then maybe release wikitcms 2.0. it may get interesting when I look at
> feeding in various possible groups of pages, but I'll burn that bridge
> when I come to it.
So I didn't quite get to the relval sub-command point yet, but I made it
a hell of a lot better today. If you want to play with my current state
of the art:
https://www.happyassassin.net/temp/tcstats_test/
is the output from a run on Fedora 20. It handles test cases that occur
multiple times both with different 'names' and with the same 'name' but
*in a different table* (it was rather badly thrown by the Desktop page
before I made wikitcms parse tables).
By default it displays the tests in more or less the order they appear
on the actual pages, with table separators - changes to the results or
tables between composes confuse it a bit, whenever a case shows up that
didn't exist in a previous compose it'll likely get stuck at the bottom
and may lead to table separator duplication.
It provides rather a lot more detail on the 'details' page: it counts
results per 'environment', lists bugs for passes and warns as well as
fails, and calculates the 'coverage' - the percentage of environments
for which a result was provided - for each page.
https://www.happyassassin.net/temp/tcstats_test/Desktop.html vs
http://testdays.qa.fedoraproject.org/testcase_stats/Category_Desktop_validation_testing.html
makes an interesting comparison (mine looked a lot like Josef's before I
taught it to parse tables) - you can see mine's more complex but also
trickable, the name of the 'release blocking' table changed after Alpha TC1,
but it's rather difficult to somehow catch that and merge the results with
those for the renamed table from later composes. The easier fix in this and
some similar cases is just to go to the wiki and make the pages consistent =)
Sorting is currently broken - that's more or less next on my list (after
somehow fixing "Install" / "Installation", which is confusing the 21
results slightly).
The current state of the code is attached for the curious, it still
needs some cleanups here and there. I made write_detail_page() a
function nested within the print_results_html() function because it
needs to be run for each test kind of in the middle of the summary page
generation loop, to make sure the links from the summary page and the
page names actually generated by the detail page writer stay consistent.
It also works as a top-level function you pass a whole *ton* of
variables into, but that seemed uglier. I might instead just make Test
class methods to generate the necessary bits and call into those, have
to try it later to see if that makes it nicer.
It still needs HEAD of wikitcms' results-types branch, I had to make
quite a lot of changes to wikitcms. All the improved result page parsing
went in there, of course.
Amongst other things, the page.ComposePage() class grew a couple of
slightly interesting new members.
from_wikipage() is a class method which you can feed an mwclient page
object (i.e. a wiki page) and it'll try and spit out the ComposePage()
object for the same page - newstats first gets all the mwclient page
objects for the release according to Release.milestone_pages(), then
feeds 'em into this method to get wikitcms page objects it can use.
sortname() is a property which returns a string that you can use to sort
ComposePage objects: for any given set of ComposePage objects, sorting
with sortname() as the key should give you the pages in the 'correct'
order, i.e. Fedora 13 Final RC1, Fedora 14 Alpha TC1, Fedora 14 Alpha
TC2, Fedora 14 Alpha RC1, Fedora 14 Beta TC1, Fedora 14 Final RC2,
Fedora 16 Alpha TC3... - the order the composes actually happened in. It
does this by join()ing the release with heavily-modified versions of the
milestone and compose that should always sort correctly...or, well, I
mean, as close as seemed reasonable to bother handling (we had some
really pretty wacky page names back before Fedora 14). newstats uses
this to sort the pages for each test type right before it gets their
result rows and turns them into newstats Test() objects. Before I made
it an instance property, I had this magnificently hideous hack instead,
which I wish to preserve for posterity:
pages.sort(key=lambda x: ' '.join((x.milestone, x.compose.replace('TC',
'AC'))))
which...actually works surprisingly well, since "Alpha", "Beta" and
"Final" happen to sort alphabetically in any case. count the
parentheses!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
#!/usr/bin/python
import wikitcms.wiki as wk
import wikitcms.event as ev
import wikitcms.page as pg
import wikitcms.release as rl
from collections import defaultdict, OrderedDict, Counter
from operator import attrgetter
import re
import os
import sys
import datetime
import shutil
import optparse
SANITY_SUB = re.compile(r'[^A-Za-z0-9]')
def get_bitmap_html(bitmap):
out = '<span style="letter-spacing:-0.5em;font-size:1.5em;">'
sym_0 = "⠄"
sym_1 = "⠄"
sym_2 = "⠆"
sym_3 = "⠇"
for compose, cnt, style in bitmap:
out += '<span class="{}">'.format(style)
if cnt == 0:
out+= sym_0
elif cnt < 3:
out+= sym_1
elif cnt < 5:
out+= sym_2
else:
out+= sym_3
out+='</span>'
out += '</span>'
return out
class Test(object):
def __init__(self):
self.passes = defaultdict(list)
self.warns = defaultdict(list)
self.fails = defaultdict(list)
self.coverage = defaultdict(int)
self.composes = set()
self.bitmap = []
self.last_tested = ""
self.p = 0
self.f = 0
self.w = 0
def update(self, compose, row):
self.milestone = row.milestone
self.composes.add(compose)
rlists = row.results.iteritems()
p = self.p
f = self.f
w = self.w
envs = 0
for env, rlist in rlists:
if rlist:
envs += 1
for result in rlist:
if result.status == 'pass':
self.last_tested = compose
self.passes[compose].append((env, result))
p += 1
if result.status == 'fail':
self.last_tested = compose
self.fails[compose].append((env, result))
f += 1
if result.status == 'warn':
self.last_tested = compose
self.warns[compose].append((env, result))
w += 1
self.coverage[compose] = envs*100 // row.boxes
self.bitmap.append([self.last_tested, 0, 'nottested'])
if p+f+w > 0:
b = self.bitmap[-1]
if f+w == 0: # just passed
b[2] = 'pass'
elif p+w == 0: # just failed
b[2] = 'fail'
elif w > 0:
b[2] = 'warn'
b[1] += p+f+w
def print_results_html(pages, tests, outdir, outfile):
# directory with templates for pages
htmldir = os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])),
'html')
col_names = ['Testcase URL','Release Level', 'Last in', 'Coverage',
'Details']
theader = "\n".join(["<th>{}</th>".format(c) for c in col_names])
trs = []
row_pre = ('<td style="padding=0"><a href="https://fedoraproject.org/'
'wiki/{testcase}">{testcase}{name}</a></td>')
row_sect = ('<tr><td style="text-align: center; font-weight: bold; '
'background-color: lightblue;" colspan="' +
str(len(col_names)) + '"><b><center>{}</b></td></tr>')
row_main = '<td style="padding=0">{}</td>\n'* (len(col_names)-2)
row_post = '<td style="padding=0"><a href="{link}.html">Details</a></td>\n'
lastsec = ''
# create detail pages
def write_detail_page():
template = open(os.path.join(htmldir, 'newtestcase.html')).read()
body = ""
for page in pages:
body += ('<p><strong><a href="https://fedoraproject.org/wiki/{p}";>'
'{p}</a></strong><br />\n').format(p=page.wikiname)
compose = (page.milestone, page.compose)
if compose not in attrs.composes:
body += "<b>No data</b>\n</p>\n"
continue
body += "<b>Coverage:</b> {}%<br/>".format(attrs.coverage[compose])
# 'g' lets us get the results for the status we're currently
# working on. results for each status are stored as tuples of the
# result environment and the result object itself.
#
#'c' counts the occurrences of each environment in the set of
# results for the status; the following line reads out the list of
# envs 'c' found and prints the name of the env and the count of
# results that was found for it.
for status in ('Passes', 'Warns', 'Fails'):
body += "<b>{}: </b>".format(status)
g = attrgetter(status.lower())
c = Counter([env for (env, res) in g(attrs)[compose]])
body += ', '.join(["{} - <b>{}</b>".format(env, c[env]) for
env in c])
body += "\n<ul>"
for (env, result) in g(attrs)[compose]:
try:
# Add a list of bugs for this status, if any.
url = ('{user}: <a href="https://bugzilla.redhat.com'
'bugzilla/show_bug.cgi?id={bug}">{bug}</a>')
body += "\n<li>" + '\n<li>'.join([url.format(
user=result.user, bug=bug) for bug in
result.bugs])
except (IndexError, TypeError):
pass
body += "\n</ul>\n"
body += "</p>\n"
fname = os.path.join(outdir, SANITY_SUB.sub('_', testcase)+SANITY_SUB.sub('_', linkname)+SANITY_SUB.sub('_', section)+'.html')
try:
fout = open(fname, 'w')
if testcase != name:
outname = ' - ' + name
else:
outname = ''
fout.write(template.format(timestamp=datetime.datetime.utcnow(),
testcase=testcase, name=outname,
body=body))
fout.close()
except:
pass
# this is where we actually start doin' stuff
for (name, testcase, section), attrs in tests.iteritems():
if section != lastsec:
trs.append(row_sect.format(section))
lastsec = section
if testcase != name:
dispname = ' - ' + name
linkname = '_' + name
else:
dispname = ''
linkname = ''
tr = row_pre.format(testcase=testcase, name=dispname)
tr += row_main.format(attrs.milestone, ' '.join(attrs.last_tested),
get_bitmap_html(attrs.bitmap))
tr += row_post.format(link=SANITY_SUB.sub('_', testcase)+SANITY_SUB.sub('_', linkname)+SANITY_SUB.sub('_', section))
trs.append("<tr>" + tr + "</tr>")
try:
write_detail_page()
except:
raise
tbody = "\n".join(trs)
# create the Overview page
print("Creating HTML output: " + os.path.join(outdir, outfile))
template = open(os.path.join(htmldir, 'index.html')).read()
fout = open(os.path.join(outdir, outfile), 'w')
fout.write(template % {'timestamp': datetime.datetime.utcnow(), 'theader': theader, 'tbody': tbody})
fout.close()
"""
wiki = wk.Wiki(('https', 'fedoraproject.org'), '/w/')
events = list()
events.append(ev.ComposeEvent('21', 'Alpha', 'TC5', wiki))
events.append(ev.ComposeEvent('21', 'Alpha', 'TC6', wiki))
events.append(ev.ComposeEvent('21', 'Alpha', 'TC7', wiki))
events.append(ev.ComposeEvent('21', 'Alpha', 'RC1', wiki))
events.append(ev.ComposeEvent('21', 'Beta', 'TC1', wiki))
events.append(ev.ComposeEvent('21', 'Beta', 'TC2', wiki))
pages = list()
for event in events:
pages.append(pg.ComposePage(wiki, event, 'Base'))
# Using an OrderedDict causes the summary page to be ordered in the same
# order as the actual result pages - except that tests that only exist for
# later composes will be at the bottom.
tests = OrderedDict()
for page in pages:
# Collections doesn't give us an OrderedDefaultDict. Boo. So, we have to
# use the old-fashioned setdefault. For each 'unique test' in the result
# rows - the combination of the test case page name and the 'test name' - a
# Test() object is created in the dict 'tests'. Each time we hit a result
# row, we call the appropriate Test()'s update method, passing it a tuple.
# The tuple's first field is another tuple that identifies the compose from
# which it came, the second field is the result row itself.
print("Processing: " + page.wikiname)
resultrows = page.get_resultrows(statuses=['pass', 'warn', 'fail'],
transferred=False)
for row in resultrows:
tests.setdefault((row.name, row.testcase), Test())
tests[(row.name, row.testcase)].update((page.event.milestone,
page.event.compose), row)
print_results_html(tests, '/tmp/tcstats_test', 'test.html')
# copy javascript
htmldir = os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), 'html')
shutil.copytree(os.path.join(htmldir, 'js'),
os.path.join('/tmp/tcstats_test', 'js'))
"""
def main():
global options
# logging.basicConfig(format='%(message)s')
# outdir = tempfile.mkdtemp(prefix = 'tc_report_')
# download and parse all pages
allpages = defaultdict(list)
wiki = wk.Wiki(('https', 'fedoraproject.org'), '/w/')
rel = rl.Release(20, wiki)
for page in rel.milestone_pages():
try:
page = pg.ComposePage.from_wikipage(page)
except:
pass
if page:
allpages[page.testtype].append(page)
for (testtype, pages) in allpages.iteritems():
tests = OrderedDict()
# sorts the pages based on a handy property wikitcms provides
pages.sort(key=attrgetter('sortname'))
for page in pages:
print("Processing: " + page.wikiname)
resultrows = page.get_resultrows(statuses=['pass', 'warn', 'fail'],
transferred=False)
for row in resultrows:
tests.setdefault((row.name, row.testcase, row.section), Test())
tests[(row.name, row.testcase, row.section)].update((page.milestone,
page.compose), row)
print_results_html(pages, tests, '/tmp/tcstats_test', testtype + '.html')
# copy javascript
htmldir = os.path.join(os.path.dirname(os.path.abspath(sys.argv[0])), 'html')
shutil.copytree(os.path.join(htmldir, 'js'), os.path.join('/tmp/tcstats_test', 'js'))
"""
in (INSTALLATION_NAME, BASE_NAME, DESKTOP_NAME):
pages = list_pages(category)
results_dict = {}
for page in pages:
try:
parse_matrix(page, results_dict)
except MatrixLookupException:
print "Failed to find Results Matrix for %s" % page
filename = SANITY_SUB.sub("_", category)
print_results_html(pages, results_dict, outdir, filename+".html")
"""
"""
# Create index page
fout = open(os.path.join(outdir, "index.html"), "w")
fout.write("<html><body>\n")
fout.write("<p>Created at: %s (UTC)</p>" % datetime.datetime.utcnow())
for category in (INSTALLATION_NAME, BASE_NAME, DESKTOP_NAME):
fname = SANITY_SUB.sub("_", category) + ".html"
fout.write("<a href=\"%s\">%s</a><br />" % (fname, category))
fout.write("\n</body></html>")
fout.close()
print "HTML Output: %s" % os.path.join(outdir, "index.html")
print outdir
"""
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
sys.exit(1)
_______________________________________________
qa-devel mailing list
[email protected]
https://admin.fedoraproject.org/mailman/listinfo/qa-devel
