On 09/03/16 21:27, Nicholas Skaggs wrote:
On 03/09/2016 04:11 PM, [email protected] wrote:
On 09/03/16 20:59, Nicholas Skaggs wrote:
On 03/09/2016 03:51 PM, [email protected] wrote:
On 09/03/16 20:43, Nicholas Skaggs wrote:
Sure. Each of the test runs are a 1 to 1 copy of a manual test.
So, in the same way you would add a result, we'll have a bot
account add a result with a Pass or Fail to the daily image. It
should also leave a comment linking to the run so you can learn
more if you are curious. Simon has actually agreed to hack on
this, so I hope we'll start to see some bot results (though they
will be failures!) on the tracker soon. We could still use some
help with fixing the actual tests however, so they can provide value!
Nicholas
I would really really not want to see anything on the tracker from
a bot - that's reporting a fail on the test not the reality :(
[snip]
Hmm, well any other opinions? You are correct at this point in that
it would be showing failures which are not true. However, we want
the tests to run and show proper pass/fails! So it should be a
temporary thing. That said, we could not post results until the
tests are working, but it's certainly possible to have a failed test
in the future that isn't a real failure.
Nicholas
The trouble with posting any fail (assuming they run properly) from
these tests to the tracker is there is absolutely no way of knowing
what failed - just gobbledygook.
So a flavour QA team would have to run the image to see if the fail
is real or not, and 'where' it failed - at that point what have we
gained?
It won't be adding a bug will it - or I would assume not.
Personally given the option to grab rss feeds from Jenkins - then
people interested in whether an image has failed could do that.
That's what I had intended way back when.
Maybe a seperate area on the tracker for them?
I was trying to keep it simple, and integrating it was the simpliest
way I could think of. A post to the notice board? Would that work better?
Nicholas
That might work - perhaps just put it there where a test fails?
So say Xubuntu failed - Notice board would say that - but say nothing
about other flavours - the expectation being we always get passes :)
--
Ubuntu-quality mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
