Re: Tested patches for openqa_fedora_tools to use fedfind and wikitcms, fix some result submissions
On Sat, 2015-02-14 at 00:55 -0800, Adam Williamson wrote:
On Fri, 2015-02-13 at 22:10 -0800, Adam Williamson wrote:
On Fri, 2015-02-13 at 22:09 -0800, Adam Williamson wrote:
On Fri, 2015-02-13 at 04:47 -0800, Adam Williamson wrote:
OK, I completed my fedfind/wikitcms conversion patch for
openqa_fedora_tools. I also fixed up some of the test case
submission data bits - they broke with relval report-auto
(instead of jskladan's version of non-interactive result
submission)
because they didn't quite grok the 'test instance' concept
report- auto follows.
Here's a follow-up (applies on top of the previous three) with
my latest work.
And here's the patch :/
Here's my latest version, just slight tweaks compared to the
previous. I discovered another of Python's pleasing sanities:
One more rev to the patch: I left a check out of that last version.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
From e49de8e778e356e0196b13fd1b59a8a67cfea37b Mon Sep 17 00:00:00 2001
From: Adam Williamson awill...@redhat.com
Date: Fri, 13 Feb 2015 19:36:12 -0800
Subject: [PATCH] further revisions to use improved fedfind/wikitcms abilities
We can still run against the latest validation event with
'current' (as before), but now we can also run arbitrary
*composes* (not just validation events) with 'compose'. It
will try and find a matching ValidationEvent and will report
results if --submit-results is passed - but if not it'll
still run the OpenQA tests against the compose. This mode
can also run without wikitcms (with only fedfind) for pure
OpenQA job initiation.
If run without args, 'compose' will run against today's
Rawhide nightly. This takes advantage of a bunch of clever
voodoo added to wikitcms and fedfind to try and ensure
version attributes are convertible freely between them,
and to guess ones that aren't supplied.
---
tools/openqa_trigger/conf_test_suites.py | 4 +-
tools/openqa_trigger/openqa_trigger.py | 141 -
tools/openqa_trigger/report_job_results.py | 2 +-
3 files changed, 100 insertions(+), 47 deletions(-)
diff --git a/tools/openqa_trigger/conf_test_suites.py b/tools/openqa_trigger/conf_test_suites.py
index fd19714..91706bd 100644
--- a/tools/openqa_trigger/conf_test_suites.py
+++ b/tools/openqa_trigger/conf_test_suites.py
@@ -96,7 +96,7 @@ TESTCASES = {
},
QA:Testcase_partitioning_guided_delete_partial: {
section: Guided storage configuration,
-env: x86,
+env: x86 BIOS,
type: Installation,
},
#: {
@@ -195,7 +195,7 @@ TESTSUITES = {
server_delete_partial: [
Server netinst,
QA:Testcase_install_to_VirtIO,
-QA:Testcase_partitioning_guided_delete_partial
+QA:Testcase_partitioning_guided_delete_partial,
QA:Testcase_Anaconda_User_Interface_Graphical,
QA:Testcase_Anaconda_user_creation,
QA:Testcase_Package_Sets_Minimal_Package_Install,
diff --git a/tools/openqa_trigger/openqa_trigger.py b/tools/openqa_trigger/openqa_trigger.py
index a6fdbe0..9a43b6c 100755
--- a/tools/openqa_trigger/openqa_trigger.py
+++ b/tools/openqa_trigger/openqa_trigger.py
@@ -8,7 +8,11 @@ import os.path
import sys
import subprocess
import argparse
-import wikitcms.wiki
+# We can at least find images and run OpenQA jobs without wikitcms
+try:
+import wikitcms.wiki
+except:
+pass
import fedfind.release
from report_job_results import report_results
@@ -33,17 +37,13 @@ def read_last():
result[version] = json_parsed.get(version, None)
return result, json_parsed
-def download_image(image, event):
+def download_image(image):
Download a given image with a name that should be unique for
this event and arch (until we start testing different images
for the same event and arch). Returns the filename of the image
(not the path).
-# We don't *really* need to pass in event and use event.version
-# here, but as we're quite strongly tied to a validation event
-# and we use it as an identifier elsewhere, using it here too
-# keeps things consistent.
-isoname = {0}_{1}.iso.format(event.version.replace(' ', '_'), image.arch)
+isoname = {0}_{1}.iso.format(image.version.replace(' ', '_'), image.arch)
filename = os.path.join(ISO_PATH, isoname)
if not os.path.isfile(filename):
# Icky hack around a urlgrabber bug:
@@ -51,13 +51,16 @@ def download_image(image, event):
urlgrabber.urlgrab(image.url.replace('https', 'http'), filename)
return isoname
-def run_openqa_jobs(isoname, arch, event_version):
+def run_openqa_jobs(isoname, arch, image_version):
# run OpenQA 'isos' job on selected isoname, with given arch
-and event version as the 'build'. **NOTE**: the 'build' is parsed
-back into the 'relval
Re: Tested patches for openqa_fedora_tools to use fedfind and wikitcms, fix some result submissions
On Fri, 2015-02-13 at 22:10 -0800, Adam Williamson wrote:
On Fri, 2015-02-13 at 22:09 -0800, Adam Williamson wrote:
On Fri, 2015-02-13 at 04:47 -0800, Adam Williamson wrote:
OK, I completed my fedfind/wikitcms conversion patch for
openqa_fedora_tools. I also fixed up some of the test case
submission data bits - they broke with relval report-auto
(instead of jskladan's version of non-interactive result
submission)
because they didn't quite grok the 'test instance' concept
report- auto follows.
Here's a follow-up (applies on top of the previous three) with my
latest work.
And here's the patch :/
Here's my latest version, just slight tweaks compared to the previous.
I discovered another of Python's pleasing sanities:
testtup = ('foo', '', 'bar')
assert '_'.join(testtup).split('_') == list(testtup)
oh python, you are awesome. so we can conveniently schlep our
release/milestone/compose tuple around with .join / .split even when
one of the values is empty; this takes advantage of that and means we
don't need the special case in report_job_results.py any more (along
with some little tweaks to make relval OK with more partial input; as
always, you need git master wikitcms and relval). just a few little
tidy-ups aside from that (and fixing another ambiguous env value in
one of the test cases).
this is a v2 of the patch in the last mail, it applies on top of the
first three.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
From e33e3819ff911eec6b6f278cf17cac53ce6b70bd Mon Sep 17 00:00:00 2001
From: Adam Williamson awill...@redhat.com
Date: Fri, 13 Feb 2015 19:36:12 -0800
Subject: [PATCH] further revisions to use improved fedfind/wikitcms abilities
We can still run against the latest validation event with
'current' (as before), but now we can also run arbitrary
*composes* (not just validation events) with 'compose'. It
will try and find a matching ValidationEvent and will report
results if --submit-results is passed - but if not it'll
still run the OpenQA tests against the compose. This mode
can also run without wikitcms (with only fedfind) for pure
OpenQA job initiation.
If run without args, 'compose' will run against today's
Rawhide nightly. This takes advantage of a bunch of clever
voodoo added to wikitcms and fedfind to try and ensure
version attributes are convertible freely between them,
and to guess ones that aren't supplied.
---
tools/openqa_trigger/conf_test_suites.py | 4 +-
tools/openqa_trigger/openqa_trigger.py | 140 -
tools/openqa_trigger/report_job_results.py | 2 +-
3 files changed, 99 insertions(+), 47 deletions(-)
diff --git a/tools/openqa_trigger/conf_test_suites.py b/tools/openqa_trigger/conf_test_suites.py
index fd19714..91706bd 100644
--- a/tools/openqa_trigger/conf_test_suites.py
+++ b/tools/openqa_trigger/conf_test_suites.py
@@ -96,7 +96,7 @@ TESTCASES = {
},
QA:Testcase_partitioning_guided_delete_partial: {
section: Guided storage configuration,
-env: x86,
+env: x86 BIOS,
type: Installation,
},
#: {
@@ -195,7 +195,7 @@ TESTSUITES = {
server_delete_partial: [
Server netinst,
QA:Testcase_install_to_VirtIO,
-QA:Testcase_partitioning_guided_delete_partial
+QA:Testcase_partitioning_guided_delete_partial,
QA:Testcase_Anaconda_User_Interface_Graphical,
QA:Testcase_Anaconda_user_creation,
QA:Testcase_Package_Sets_Minimal_Package_Install,
diff --git a/tools/openqa_trigger/openqa_trigger.py b/tools/openqa_trigger/openqa_trigger.py
index a6fdbe0..d8e6efc 100755
--- a/tools/openqa_trigger/openqa_trigger.py
+++ b/tools/openqa_trigger/openqa_trigger.py
@@ -8,7 +8,11 @@ import os.path
import sys
import subprocess
import argparse
-import wikitcms.wiki
+# We can at least find images and run OpenQA jobs without wikitcms
+try:
+import wikitcms.wiki
+except:
+pass
import fedfind.release
from report_job_results import report_results
@@ -33,17 +37,13 @@ def read_last():
result[version] = json_parsed.get(version, None)
return result, json_parsed
-def download_image(image, event):
+def download_image(image):
Download a given image with a name that should be unique for
this event and arch (until we start testing different images
for the same event and arch). Returns the filename of the image
(not the path).
-# We don't *really* need to pass in event and use event.version
-# here, but as we're quite strongly tied to a validation event
-# and we use it as an identifier elsewhere, using it here too
-# keeps things consistent.
-isoname = {0}_{1}.iso.format(event.version.replace(' ', '_'), image.arch)
+isoname = {0}_{1}.iso.format(image.version.replace(' ', '_'), image.arch)
filename =
Tested patches for openqa_fedora_tools to use fedfind and wikitcms, fix some result submissions
OK, I completed my fedfind/wikitcms conversion patch for openqa_fedora_tools. I also fixed up some of the test case submission data bits - they broke with relval report-auto (instead of jskladan's version of non-interactive result submission) because they didn't quite grok the 'test instance' concept report-auto follows. Note this needs the very latest git wikitcms. I don't think latest relval or fedfind is strictly required, but may as well bump them anyway. I've installed the current wikitcms on fedora-qa-01. I will cut new releases tomorrow. I have tested this on fedora-qa-01, it ran and submitted a full set of results for 2015-02-07. I believe it should work for Branched nightlies (if we wind up nominating any) and TC/RC builds also, which is a significant win. I did test this a bit: https://fedora-qa-01.lab.bos.redhat.com/tests/15 is a one-job run against F21 Final RC5, you can see it successfully got the ISO and started the test but the test fails because the 'Timbuktu' screen isn't there. I revised the openqa_trigger.py CLI to use argparse; it has the same basic modes as before but the syntax is a bit different. 'openqa_trigger.py current' runs against the 'current' compose event if it has not already done so (what just plain 'openqa_trigger.py' did before). 'openqa_trigger.py event' takes the same 'version id' arguments as 'relval report-auto' and runs tasks (but does not automatically report results) against the specified compose (what you used to get by passing args, approximately, but much more robust and capable). The basic approach is that openqa_trigger gets a ValidationEvent from python-wikitcms - either the Wiki.current_event property for 'current', or the event specified, obtained via the newly-added Wiki.get_validation_event(), for 'event'. For 'event' it then just goes ahead and runs the jobs and prints the IDs. For 'current' it checks the last run compose version for each arch and runs if needed, as before. The ValidationEvent's 'sortname' property is the value written out to PERSISTENT to track the 'last run' - this property is intended to always sort compose events 'correctly', so we should always run when appropriate even when going from Rawhide to Branched, Branched to a TC, TC to RC, RC to (next milestone) TC. On both paths it gets a fedfind.Release object via the ValidationEvent - ValidationEvents have a ff_release property which is the fedfind.Release object that matches that event. It then queries fedfind for image locations using a query that tries to get just *one* generic-ish network install image for each arch. It passes the location to download_image(), which is just download_rawhide_iso() renamed and does the same job, only it can be simpler now. From there it works pretty much as before, except we use the ValidationEvent's 'version' property as the BUILD setting for OpenQA, and report_job_results get_relval_commands() is tweaked slightly to parse this properly to produce a correct report-auto command. Probably the most likely bits to break here are the sortname thing (see wikitcms helpers.py fedora_release_sort(), it's pretty stupid, I should re-write it) and the image query, which might wind up getting more than one image depending on how exactly the F22 Alpha composes look. I'll keep a close eye on that. We can always take the list from fedfind and further filter it so we have just one image per arch. Image objects have a .arch attribute so this will be easy to do if necessary. I *could* give the fedfind query code a 'I'm feeling lucky'- ish mode to only return one image per (whatever), but not sure if that would be too specialized, I'll think about it. Please do ask any questions :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net From 55d5d1b4ce4bfd7ae7c239c8aca03bceee2f1f06 Mon Sep 17 00:00:00 2001 From: Adam Williamson awill...@redhat.com Date: Thu, 12 Feb 2015 19:32:27 -0800 Subject: [PATCH 1/3] use python-wikitcms and fedfind to handle events and images This removes a bunch of template-y bits that fedfind does better, and should make things work for all validation events. The CLI for openqa_trigger is completely redone to use argparse and check values and stuff. The syntax for the event sub-command is purposely similar to relval report-auto. Note this requires wikitcms changes currently in git master. --- tools/openqa_trigger/openqa_trigger.py | 159 +++-- tools/openqa_trigger/report_job_results.py | 7 +- 2 files changed, 108 insertions(+), 58 deletions(-) diff --git a/tools/openqa_trigger/openqa_trigger.py b/tools/openqa_trigger/openqa_trigger.py index 9406d0c..7424c8b 100755 --- a/tools/openqa_trigger/openqa_trigger.py +++ b/tools/openqa_trigger/openqa_trigger.py @@ -7,15 +7,15 @@ import urlgrabber import os.path import sys import subprocess +import argparse +import
Re: Tested patches for openqa_fedora_tools to use fedfind and wikitcms, fix some result submissions
On Fri, 2015-02-13 at 04:47 -0800, Adam Williamson wrote: OK, I completed my fedfind/wikitcms conversion patch for openqa_fedora_tools. I also fixed up some of the test case submission data bits - they broke with relval report-auto (instead of jskladan's version of non-interactive result submission) because they didn't quite grok the 'test instance' concept report-auto follows. Here's a follow-up (applies on top of the previous three) with my latest work. This revises the openqa bits to take advantage of all the consolidation of version handling across wikitcms/relval/fedfind. This gives us some cool advantages for openqa. The 'current' command works just as before, basically. All the fun is in 'compose'. compose now basically just asks fedfind for images for the release you specify - there's lots of magic guessing now, so you can be sloppy with what you pass to some extent (calling it with no parameters at all tries to run on nightlies for today's date, for e.g.). If you pass --submit-results it does a sanity check to see if it can find a matching wikitcms validation event and disables result submission if not (this isn't really strictly *necessary*, but it seems polite). If it does find a matching validation event and you pass --submit-results, it'll go ahead and submit results at the end. In 'compose', all wikitcms use is optional - you should be able to use it without wikitcms at all, and it'll still run the OpenQA jobs. Both 'current' and 'compose' now use the fedfind.Image.version attribute for identification. For all composes except Rawhide nightlies, this should always be a 'RELEASE MILESTONE COMPOSE' string which we can parse right back into relval parameters, just as with the wikitcms.ValidationEvent.version attribute from the previous commit. For Rawhide nightlies it will be 'Rawhide COMPOSE' (with COMPOSE as a date, obviously), and there's a special case in report_job_results.py to handle that - bit inelegant, but it'll do for now. With this you can absolutely trivially have a bot that runs on Rawhide every day: it should just run 'openqa_trigger.py compose' every day at some point after the compose is usually finished. A bot that tries to run on Branched every day is a bit trickier, because fedfind can't guess Branched. The way I've hooked everything up we can't really have wikitcms do it either, because I made wikitcms' guessing only guess events that exist (which makes sense for wikitcms). We could put a quick hack in openqa_trigger which tries to get the current release from the Wiki and uses that, I'll think about how hacky that would be. Otherwise of course you just have to write a cronjob that passes --release and remember to update it with each new Branched release. Hope this is useful! I'm thinking about whether it's worth setting up a special category for non-nominated nightly composes on the wiki for long-term storage of Coconut results for non-nominated nightlies - it wouldn't be too hard to add such a thing to wikitcms and have openqa report to it. But maybe we can go a bit slow and see how it goes for now. I'd also like to get into the 'needle' stuff and try to make it do useful things on failure, like running through the crash submission procedure when it hits a crash, then filing a 'fail' report linked to the bug. This version of openqa_fedora_tools, along with the latest git fedfind and wikitcms and relval that are needed to use it, is in place on fedora-qa-01 for those with access. Maybe I'll set up some cronjobs on qa-01 tomorrow with a FAS account called 'colada'...:) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
