Re: [Tails-dev] Testing with openqa?
hi,
anonym wrote (05 Oct 2015 15:33:55 GMT) :
> On 09/15/2015 11:03 AM, intrigeri wrote:
>> I've had a look during DebConf, and indeed the web interface for
>> developers is much better than what Jenkins will give us as-is.
>> Perhaps at some point we'll need $something that extracts data from
>> Jenkins and presents it to developers, and at that point it may be
>> worth looking into OpenQA.
> I'm unsure what the "$something that extracts data from Jenkins and
> presents it to developers" part refers to.
I meant that some day, we might want to give Tails contributors
something more usable than the Jenkins web UI, as their primary
interface to our automated QA systems. But nowadays most contributors
don't even have access to our Jenkins' web UI, so we're not exactly
there yet.
> If it refers to Alan's "it has a nice web interface to update
> screenshots", then [...]
JFTR: I wasn't referring to that, and I agree it's not useful to us.
>> Note that tests are written directly in Perl, with no overlay language
>> like Gherkin. Not as if we were taking much benefit from Gherkin for
>> {intra,inter}-team communication yet, but I like it that we're able to;
>> and I intend to try to use Gherkin more in the future when discussing
>> changes with UX folks.
> I personally like the clarity of what is (well, *should be*) going on
> that they provide to me, as a developer, even without the "easier
> communication with non-developers" part.
Me too, but as you're suggesting this is quite theoretical currently,
e.g. given the diversity of abstraction levels we use in there.
Cheers!
--
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Testing with openqa?
On 09/15/2015 11:03 AM, intrigeri wrote:
> Alan wrote (23 Aug 2015 13:28:41 GMT) :
>> but I encourage the people working on the test suite to have a look
>> at OpenQA and consider wether it would make sense to use it. It has
>> a nice web interface to update screenshots,
Sorry for not responding earlier!
I looked at OpenQA ~4 years ago, before we started our own thing, but it
seemed to me that it solved quite little of what we actually needed, at
least on the top; it didn't support fine-grained hardware configuration
(and not hotplugging at all), for instance. I felt weary of using
something that we'd have to extend endlessly to make it useful for us,
and while it would have been a potentially better longer-term solution,
I'm not sure we'd have started to benefit from it even now. (Also, perl!)
When bertagaz made the initial version of the cucumber + sikuli +
libvirt based thing we have now, I was blown away at how easily
extensible it was (despite me having no previous experience with Ruby).
I still think that is true, despite some painful surprises (JRuby). We
need so much flexibility to do all the things we want to do, so I'm not
sure if anything else than a solution of our own would work efficiently.
Or maybe I'm just having a case of NIH syndrome. :)
>> and we would share maintenance with other distros.
This I find really sad, however.
> I've had a look during DebConf, and indeed the web interface for
> developers is much better than what Jenkins will give us as-is.
> Perhaps at some point we'll need $something that extracts data from
> Jenkins and presents it to developers, and at that point it may be
> worth looking into OpenQA.
I'm unsure what the "$something that extracts data from Jenkins and
presents it to developers" part refers to. If it refers to Alan's "it
has a nice web interface to update screenshots", then I don't see the
benefit in it. It's quite rare that only pictures need updating when
stuff changes, and when they do, often more than one (used in the same
scenario) needs updating. Our --retry-find option will let us solve that
in (ideally) one babysitted run, while relying on automation + a web gui
to update images could result in some crazy long back and forth, or that
you must run a VM in parallel to take screenshots from. I'm not
convinced we'll have much use of that feature except for trivial cases
where only one image needs updating. And in trivial cases we can just
extract the new image from the test suite failure screenshot.
> Note that tests are written directly in Perl, with no overlay language
> like Gherkin. Not as if we were taking much benefit from Gherkin for
> {intra,inter}-team communication yet, but I like it that we're able to;
> and I intend to try to use Gherkin more in the future when discussing
> changes with UX folks.
I personally like the clarity of what is (well, *should be*) going on
that they provide to me, as a developer, even without the "easier
communication with non-developers" part.
Cheers!
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Testing with openqa?
sajolida:
>>> Not as if we were taking much benefit from Gherkin for
>>> {intra,inter}-team communication yet, but I like it that we're able to;
>>> and I intend to try to use Gherkin more in the future when discussing
>>> changes with UX folks.
>>
>> This sounds interesting. I have read some negative things about this
>> but the argument made was that the language did exactly as it
>> advertised; making humans still have to talk to other humans, as it
>> didn't magically resolve their socially retarded nature :)
>>
>> I am open to poking around with this sooner than later, even in test
>> communication runs, if not with live designations decisions.
>>
>> I have copied the UX list in case people want to nerd out over there, too.
>
> Yeap, I'd like to know more about that intrigeri meant but I guess that
> will be done in due time.
I recommend you tails-ux@ people to look at the features/*.feature files
in Tails' Git. Ideally anyone with domain knowledge about *using* Tails
following the steps in those files should be able to reproduce more or
less exactly what our automated test suite does when following them. No
further explanation should be needed, and in particular one shouldn't
have to look at the code == step definitions.
That said, in many cases this is not true in its current state due to
technical limitations/shortcuts (but there will be quite a few nice
improvements soon, with #6094 being solved). But it would be interesting
to get some help with #10329 from "outsiders" like you tails-ux@ people.
I suppose we, as Test Suite Developers, often think a bit too much about
the implementation, resulting in scenarios where steps are split
according to that, contrary to how a user might think. That may actually
be a good thing (to reduce the code complexity for the step definitions)
but it would nonetheless be interesting to see how they would compare.
Perhaps there is a better balance.
Cheers!
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Testing with openqa?
spencer...@openmailbox.org:
>> intrigeri:
>> Note that tests are written directly in Perl, with no overlay language
>> like Gherkin.
>
> Where can I learn more about testing?
I think that they are talking here about the automated test suite. See
https://tails.boum.org/contribute/release_process/test/automated_tests.
>> Not as if we were taking much benefit from Gherkin for
>> {intra,inter}-team communication yet, but I like it that we're able to;
>> and I intend to try to use Gherkin more in the future when discussing
>> changes with UX folks.
>
> This sounds interesting. I have read some negative things about this
> but the argument made was that the language did exactly as it
> advertised; making humans still have to talk to other humans, as it
> didn't magically resolve their socially retarded nature :)
>
> I am open to poking around with this sooner than later, even in test
> communication runs, if not with live designations decisions.
>
> I have copied the UX list in case people want to nerd out over there, too.
Yeap, I'd like to know more about that intrigeri meant but I guess that
will be done in due time.
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
[Tails-dev] Testing with openqa?
Hi,
intrigeri:
Note that tests are written directly in Perl, with no overlay language
like Gherkin.
Where can I learn more about testing?
Not as if we were taking much benefit from Gherkin for
{intra,inter}-team communication yet, but I like it that we're able to;
and I intend to try to use Gherkin more in the future when discussing
changes with UX folks.
This sounds interesting. I have read some negative things about this
but the argument made was that the language did exactly as it
advertised; making humans still have to talk to other humans, as it
didn't magically resolve their socially retarded nature :)
I am open to poking around with this sooner than later, even in test
communication runs, if not with live designations decisions.
I have copied the UX list in case people want to nerd out over there,
too.
Wordlife,
Spencer
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Testing with openqa?
Alan wrote (23 Aug 2015 13:28:41 GMT) :
> but I encourage the people working on the test suite to have a look
> at OpenQA and consider wether it would make sense to use it. It has
> a nice web interface to update screenshots, and we would share
> maintenance with other distros.
I've had a look during DebConf, and indeed the web interface for
developers is much better than what Jenkins will give us as-is.
Perhaps at some point we'll need $something that extracts data from
Jenkins and presents it to developers, and at that point it may be
worth looking into OpenQA.
Note that tests are written directly in Perl, with no overlay language
like Gherkin. Not as if we were taking much benefit from Gherkin for
{intra,inter}-team communication yet, but I like it that we're able to;
and I intend to try to use Gherkin more in the future when discussing
changes with UX folks.
Cheers,
--
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
[Tails-dev] Testing with openqa?
Hi, At GUADEC I talked with SuSE people about testing. They use something called OpenQA[1] that looks like a software that does basically what our automated test suite does[2]: test installers, live isos and applications integrations by clicking on UI images in VMs. They are trying to convince other distributions to take benefit on the work they did on it. Too bad we did develop two solutions in parallel... but I encourage the people working on the test suite to have a look at OpenQA and consider wether it would make sense to use it. It has a nice web interface to update screenshots, and we would share maintenance with other distros. Cheers 1. https://os-autoinst.github.io/openQA/ 2. https://en.opensuse.org/openSUSE:OpenQA ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
