Re: [Tails-dev] Releasing automated tests

[intrigeri]

Hi,

bertagaz wrote (03 Dec 2015 15:40:09 GMT) :
> On Tue, Dec 01, 2015 at 09:28:13PM +0100, intrigeri wrote:
>>  * the list of scenarios that shall be tagged @doc: at least the one
>>anonym mentioned earlier in this thread; plus "check all PO files"
>>I guess; the WhisperBack tests once they have been written; more?

> I've added a ticket (#10706) and a branch containing what I've found to
> be relevant to be tagged as such.

Cool! Merged, and added a note on #10270 :)

>>  * some process to make sure we tag scenarios @doc whenever relevant;
>>I think we'll forget from time to time, and it's fine, because
>>we'll be adding such scenarios very rarely; so perhaps a yearly
>>check by the RM (e.g. 1st release of the year) is good enough?

> Added that to the RM role in commit 4b176e2.

Thanks. I felt that the room for interpretation (and for skipping this
task forever) the phrasing left was way too big so I made it much
smaller ⇒ please review recent changes I did on that page.

>>  * and finally, some Redmine tickets describing this decision and the
>>timeline of its implementation.

> I've also added #10707 which is about the actual implementation of this
> feature. It has a dedicated branch with a commit doing so.

Replied on the ticket, let's follow-up there.

> I don't see what other tickets we may want to add for this.

OK (⇒ closed #10492! :)

Cheers,
-- 
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[intrigeri]

Hi,

[too bad it's too late to rename this thread, now that there have been
6 replies, to it.]

anonym wrote (21 Oct 2015 12:46:37 GMT) :

>> FTR I dislike the idea of blacklisting such branches based on their
>> name. I'm not going to debate it here [...]

> Please do it here, then! :) I guess it's related to that you want at
> least doc/ branches to be tested, since our automated test suite
> actually depend on the documentation (see two quotes below for more on
> this).

My concerns about skipping branches based on their name are
essentially the same as the ones I've just posted in another
sub-thread of this one
(https://mailman.boum.org/pipermail/tails-dev/2015-December/009836.html).

Besides, I think they are addressed by the content-based filtering (as
opposed to name-based filtering) that seems to be the most current
proposal in this thread, so no need for me to elaborate about it now.

> Ok, let's forget about this now and return to the "workaround", i.e.
> skipping tests depending on branch name:

>> Still, this idea is probably OK as a short-term workaround if we need
>> it for some reason, but then we need a clear description of the
>> problem,

> The problem, as I see it, is: we run the full automated test suite for
> doc/ and web/ branches, which wastes a lot of valuable isotester time
> given that all but a few tests are completely orthogonal to changes in
> the wiki sub directory of Tails Git sources.

About isotester time, I computed some stats in order to make my
opinion a bit better informed: in November, among 314 non-aborted ISO
test suite runs, 24% (75) were about branches whose name starts with
"doc" or "web". So yes, that's wasting a lot of resources. It will
probably be a problem, we'll see on #9264, so all the work put into
this thread is definitely useful :)

It seems that the proposed solution also addresses the problem that
was raised initially, which was rather about latency for testing
non-doc branches, so it sounds like we can avoid introducing clever
stuff like prioritization + quiet time for a bit longer (#9760 will
likely need it anyway, though). Yay!

>> [...] making sure that the workaround is not worst than the problem
>> it fixes

> The only effect should be that we won't get automated tests of the few
> scenarios that looks at the wiki shipped inside Tails. I think that
> currently is one scenario: The Report an Error launcher will open the
> support documentation in supported non-English locales.

> So, if some wiki
> change makes the 'the support documentation page opens in Tor Browser'
> step fail. The only way that can happen is if:
> [...]
> Only in the first case would our test suite find an actual error, the
> other two just implies that our test is now out of sync and must be
> updated. Clearly it's not worth wasting that much isotester time on this
> at the moment.

Thanks for gathering all the data I need to be convinced :)

I'll comment on this topic, about the actual proposed solution, later
on this thread if needed.

Cheers,
-- 
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[intrigeri]

Hi,

anonym wrote (28 Oct 2015 16:52:31 GMT) :
> Then I think we can combine the "..." operator with another fancy Git
> feature I recently found, namely Git pathspec "magic signatures". So we
> could do:

>BASE_BRANCH_DIFF="$(git diff $base_branch...$commit  -- \
>'*' \
>':!/wiki' \
>':!/ikiwiki.setup' \
>':!/ikiwiki-cgi.setup')"
>if [ -z "${BASE_BRANCH_DIFF}" ]; then
>CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc"
>fi

> where $commit is the commit we test, before merging the base branch
> locally. Interesting!

So, it seems that the current proposal is to skip the temporary
workaround (name-based filtering), and instead directly do
content-based filtering. Fine by me!

What's missing is:

 * the list of scenarios that shall be tagged @doc: at least the one
   anonym mentioned earlier in this thread; plus "check all PO files"
   I guess; the WhisperBack tests once they have been written; more?

 * some process to make sure we tag scenarios @doc whenever relevant;
   I think we'll forget from time to time, and it's fine, because
   we'll be adding such scenarios very rarely; so perhaps a yearly
   check by the RM (e.g. 1st release of the year) is good enough?

 * and finally, some Redmine tickets describing this decision and the
   timeline of its implementation.

bertagaz: once we have all that, please reassign #10492 to me for
review.

Cheers,
-- 
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[intrigeri]

Hi,

sajolida wrote (22 Oct 2015 12:19:04 GMT) :
>>> FTR I dislike the idea of blacklisting such branches based on their
>>> name. I'm not going to debate it here [...]

> The prefixes doc/ and web/ are used loosely to differentiate work on the
> "documentation" (/doc /support) and the "website" in general (structure,
> stuff not in /doc, etc.) but the difference is not strict.

Indeed. I've also seen the "contrib/", "about/" and "faq/" prefixes
being used. Branch naming is quite arbitrary, and sometimes creative,
so yay, instead of relying on a hard-coded list of prefixes:

> I also don't think they should be tested. Maybe you could exclude them
> from testing by their diff to their base branch. If all the diff is
> under wiki/src then don't test that branch.

I like the idea!

Cheers,
-- 
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[bertagaz]

Hi,

On Wed, Nov 18, 2015 at 02:38:00PM +0100, intrigeri wrote:
> bertagaz wrote (18 Nov 2015 12:34:34 GMT) :
> > Intrigeri, what's on your opinion on [...]
> 
> Please give me an explicit deadline.

I probably won't be able to work on it before week 49 anyway, so no
rush, but an answer before would be cool.

> Thanks for asking!

Well, you express a strong blocking position, can't get a real consensus
without an update from you :)

Bert.
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[bertagaz]

Hi,

Sorry for the late reply.

On Wed, Oct 28, 2015 at 04:52:31PM +, anonym wrote:
> sajolida:
> > anonym:
> >> sajolida:
> > From: intrigeri 
> > Date: Tue, 20 Oct 2015 13:08:31 +0200
> >
> > FTR I dislike the idea of blacklisting such branches based on their
> > name. I'm not going to debate it here [...]

Intrigeri, what's on your opinion on the proposed way to blacklist them
made in this thread by anonym and sajolida (see below)? It's not based
on their name anymore.

> >>> I also don't think they should be tested. Maybe you could exclude them
> >>> from testing by their diff to their base branch. If all the diff is
> >>> under wiki/src then don't test that branch.

That seems to be the consensus here then, unless someone else raises.

> Then I think we can combine the "..." operator with another fancy Git
> feature I recently found, namely Git pathspec "magic signatures". So we
> could do:
> 
>BASE_BRANCH_DIFF="$(git diff $base_branch...$commit  -- \
>'*' \
>':!/wiki' \
>':!/ikiwiki.setup' \
>':!/ikiwiki-cgi.setup')"
>if [ -z "${BASE_BRANCH_DIFF}" ]; then
>CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc"
>fi
> 
> where $commit is the commit we test, before merging the base branch
> locally. Interesting!

Agree, I like this way to encode it. Sounds at least like a nice baby
step to begin with, simple and effective.

This whole thread remembered me the discussion we had earlier about
feature branches, and made me think (again) we could as well use the
cucumber tag facility to lighten the number of tests: if building from a
feature|test branch, run with the --tag $branch_name, so developers can
choose which test they want their branch to be tested against. May be
useful for some branches were we know that it'll break some tests. But
we dropped this idea.

So if everyone agree on the proposal made here by anonym and sajolida, I
guess that's now a new "Code" ticket.

bert.
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[intrigeri]

bertagaz wrote (18 Nov 2015 12:34:34 GMT) :
> Intrigeri, what's on your opinion on [...]

Please give me an explicit deadline.
Thanks for asking!

___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[sajolida]

anonym:
> sajolida:
>> anonym:
>>> [Moving discussion to tails-dev@]
>>
>> Meta: I really don't want to understand everything that's in this email
>> but I felt you would want me to answer this one. But if you think that
>> you can have this discussion without me I would be super happy as well.
> 
> I believe you have answered the question that was (mostly) directed to
> you, but you also added an interesting idea, so... :)
> 
>>> Given the trimming that has happened, some context may have been lost.
>>> The discussion is about that we now, in our Jenkins setup, automatically
>>> test images built from doc/ and web/ branches, which wastes a lot of
>>> time on our isotesters.
>>>
 From: intrigeri 
 Date: Tue, 20 Oct 2015 13:08:31 +0200

> From: bertagaz 
> Date: Mon, 19 Oct 2015 12:29:26 +0200

>> From: anonym 
>
>> Still, once we release 1.7, then all doc/ and web/ branches will become
>> tested. I suspect we will need a permanent fix for only building (not
>> testing) these branches -- it's useless to test them 99.9% of the time,
>> and they will block (for ~5 hours) test runs from relevant branches that
>> got something pushed to them right after them.

> That's something we didn't decide when during the design round. Sounds
> doable, but I wonder if there are still some valid points to still test
> that branches.
>>>
>>> True, but there's an overwhelming amount of them, and their
>>> modifications are limited to something that is completely isolated from
>>> most of Tails, the OS, meaning that a huge part of each test run on them
>>> is just a (possibly out-dated) re-run of master/devel/stable depending
>>> on which branch it was based on. That is unlike feature/, bugfix/ and
>>> test/ branches, that need a full run in general. Perhaps you can see
>>> where I'm going:
>>>
>>> As an optimization, we could introduce @web and @doc tags, and run the
>>> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
>>> respectively. Then we could even have automated tests of @web changes
>>> before deploying them by browsing the local wiki in Tails. :)
>>>
>>> Note that I may not have the correct understanding of the doc/ vs web/
>>> distinction, so I'd like a clarification just in case so we don't design
>>> something stupid. I suspect that since we don't have any automated tests
>>> for the *website* (as opposed to the docs) we only care about doc/ and
>>> only need to test web/ if we want to start testing the website.
>>>
 FTR I dislike the idea of blacklisting such branches based on their
 name. I'm not going to debate it here [...]
>>
>> The prefixes doc/ and web/ are used loosely to differentiate work on the
>> "documentation" (/doc /support) and the "website" in general (structure,
>> stuff not in /doc, etc.) but the difference is not strict.
> 
> ACK, as I expected, than.
> 
>> I also don't think they should be tested. Maybe you could exclude them
>> from testing by their diff to their base branch. If all the diff is
>> under wiki/src then don't test that branch.
> 
> I guess you mean the diff against the base branch (but base branches
> themselves would *always* build, of course). Hm. Technically we'd have
> to do something slightly different since a `git diff` would show changes
> in the base branch since the point they diverged. We'd have to look at
> all files touches in `git log $base_branch..` or something like that.
> It's an interesting approach, which I think I like.

Did you took into account the '...' (THREE DOTS) operator which is
slightly different than '..' and I *think* might be helpful here to diff
only the changes that happened on this branch. I'm not sure what it does
exactly (couldn't understand the man page).

 [...] making sure that the workaround is not worst than the problem
 it fixes
>>>
>>> The only effect should be that we won't get automated tests of the few
>>> scenarios that looks at the wiki shipped inside Tails. [...]
>>
>> Agreed. If I understand correctly, these scenarios have a *dependency*
>> on what is on the local copy of the website, but they are actually
>> testing the Tails software (the "Report an Error" launcher for example).
> 
> Correct so far.
> 
>> They are not testing the content of the local copy of the website itself.
> 
> Actually, they *are* testing the content of the local copy, e.g. that
> the support page is properly localized. Hence there could be a subset of
> automated tests that would make sense to run for doc/ and web/ branches.

I didn't know that, sorry!
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[anonym]

sajolida:
> anonym:
>> sajolida:
>>> anonym:
 [Moving discussion to tails-dev@]
>>>
>>> Meta: I really don't want to understand everything that's in this email
>>> but I felt you would want me to answer this one. But if you think that
>>> you can have this discussion without me I would be super happy as well.
>>
>> I believe you have answered the question that was (mostly) directed to
>> you, but you also added an interesting idea, so... :)
>>
 Given the trimming that has happened, some context may have been lost.
 The discussion is about that we now, in our Jenkins setup, automatically
 test images built from doc/ and web/ branches, which wastes a lot of
 time on our isotesters.

> From: intrigeri 
> Date: Tue, 20 Oct 2015 13:08:31 +0200
>
>> From: bertagaz 
>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>
>>> From: anonym 
>>
>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>> tested. I suspect we will need a permanent fix for only building (not
>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>> and they will block (for ~5 hours) test runs from relevant branches that
>>> got something pushed to them right after them.
>
>> That's something we didn't decide when during the design round. Sounds
>> doable, but I wonder if there are still some valid points to still test
>> that branches.

 True, but there's an overwhelming amount of them, and their
 modifications are limited to something that is completely isolated from
 most of Tails, the OS, meaning that a huge part of each test run on them
 is just a (possibly out-dated) re-run of master/devel/stable depending
 on which branch it was based on. That is unlike feature/, bugfix/ and
 test/ branches, that need a full run in general. Perhaps you can see
 where I'm going:

 As an optimization, we could introduce @web and @doc tags, and run the
 test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
 respectively. Then we could even have automated tests of @web changes
 before deploying them by browsing the local wiki in Tails. :)

 Note that I may not have the correct understanding of the doc/ vs web/
 distinction, so I'd like a clarification just in case so we don't design
 something stupid. I suspect that since we don't have any automated tests
 for the *website* (as opposed to the docs) we only care about doc/ and
 only need to test web/ if we want to start testing the website.

> FTR I dislike the idea of blacklisting such branches based on their
> name. I'm not going to debate it here [...]
>>>
>>> The prefixes doc/ and web/ are used loosely to differentiate work on the
>>> "documentation" (/doc /support) and the "website" in general (structure,
>>> stuff not in /doc, etc.) but the difference is not strict.
>>
>> ACK, as I expected, than.
>>
>>> I also don't think they should be tested. Maybe you could exclude them
>>> from testing by their diff to their base branch. If all the diff is
>>> under wiki/src then don't test that branch.
>>
>> I guess you mean the diff against the base branch (but base branches
>> themselves would *always* build, of course). Hm. Technically we'd have
>> to do something slightly different since a `git diff` would show changes
>> in the base branch since the point they diverged. We'd have to look at
>> all files touches in `git log $base_branch..` or something like that.
>> It's an interesting approach, which I think I like.
> 
> Did you took into account the '...' (THREE DOTS) operator which is
> slightly different than '..' and I *think* might be helpful here to diff
> only the changes that happened on this branch. I'm not sure what it does
> exactly (couldn't understand the man page).

Neat! That's is a very useful Git feature. Thanks for letting me know
about it!

Then I think we can combine the "..." operator with another fancy Git
feature I recently found, namely Git pathspec "magic signatures". So we
could do:

   BASE_BRANCH_DIFF="$(git diff $base_branch...$commit  -- \
   '*' \
   ':!/wiki' \
   ':!/ikiwiki.setup' \
   ':!/ikiwiki-cgi.setup')"
   if [ -z "${BASE_BRANCH_DIFF}" ]; then
   CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc"
   fi

where $commit is the commit we test, before merging the base branch
locally. Interesting!

Cheers!

___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

[Tails-dev] Releasing automated tests

[Spencer]

Hi,



anonym:
Neat! That's is a very useful Git feature. Thanks for letting me know
about it!

Then I think we can combine the "..." operator with another fancy Git
feature I recently found, namely Git pathspec "magic signatures". So we
could do:

   BASE_BRANCH_DIFF="$(git diff $base_branch...$commit  -- \
   '*' \
   ':!/wiki' \
   ':!/ikiwiki.setup' \
   ':!/ikiwiki-cgi.setup')"
   if [ -z "${BASE_BRANCH_DIFF}" ]; then
   CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc"
   fi

where $commit is the commit we test, before merging the base branch
locally. Interesting!



I don't understand this at all!

Don't mind me :)

Wordlife,
Spencer

___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[anonym]

sajolida:
> anonym:
>> [Moving discussion to tails-dev@]
> 
> Meta: I really don't want to understand everything that's in this email
> but I felt you would want me to answer this one. But if you think that
> you can have this discussion without me I would be super happy as well.

I believe you have answered the question that was (mostly) directed to
you, but you also added an interesting idea, so... :)

>> Given the trimming that has happened, some context may have been lost.
>> The discussion is about that we now, in our Jenkins setup, automatically
>> test images built from doc/ and web/ branches, which wastes a lot of
>> time on our isotesters.
>>
>>> From: intrigeri 
>>> Date: Tue, 20 Oct 2015 13:08:31 +0200
>>>
 From: bertagaz 
 Date: Mon, 19 Oct 2015 12:29:26 +0200
>>>
> From: anonym 

> Still, once we release 1.7, then all doc/ and web/ branches will become
> tested. I suspect we will need a permanent fix for only building (not
> testing) these branches -- it's useless to test them 99.9% of the time,
> and they will block (for ~5 hours) test runs from relevant branches that
> got something pushed to them right after them.
>>>
 That's something we didn't decide when during the design round. Sounds
 doable, but I wonder if there are still some valid points to still test
 that branches.
>>
>> True, but there's an overwhelming amount of them, and their
>> modifications are limited to something that is completely isolated from
>> most of Tails, the OS, meaning that a huge part of each test run on them
>> is just a (possibly out-dated) re-run of master/devel/stable depending
>> on which branch it was based on. That is unlike feature/, bugfix/ and
>> test/ branches, that need a full run in general. Perhaps you can see
>> where I'm going:
>>
>> As an optimization, we could introduce @web and @doc tags, and run the
>> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
>> respectively. Then we could even have automated tests of @web changes
>> before deploying them by browsing the local wiki in Tails. :)
>>
>> Note that I may not have the correct understanding of the doc/ vs web/
>> distinction, so I'd like a clarification just in case so we don't design
>> something stupid. I suspect that since we don't have any automated tests
>> for the *website* (as opposed to the docs) we only care about doc/ and
>> only need to test web/ if we want to start testing the website.
>>
>>> FTR I dislike the idea of blacklisting such branches based on their
>>> name. I'm not going to debate it here [...]
> 
> The prefixes doc/ and web/ are used loosely to differentiate work on the
> "documentation" (/doc /support) and the "website" in general (structure,
> stuff not in /doc, etc.) but the difference is not strict.

ACK, as I expected, than.

> I also don't think they should be tested. Maybe you could exclude them
> from testing by their diff to their base branch. If all the diff is
> under wiki/src then don't test that branch.

I guess you mean the diff against the base branch (but base branches
themselves would *always* build, of course). Hm. Technically we'd have
to do something slightly different since a `git diff` would show changes
in the base branch since the point they diverged. We'd have to look at
all files touches in `git log $base_branch..` or something like that.
It's an interesting approach, which I think I like.

>>> [...] making sure that the workaround is not worst than the problem
>>> it fixes
>>
>> The only effect should be that we won't get automated tests of the few
>> scenarios that looks at the wiki shipped inside Tails. [...]
> 
> Agreed. If I understand correctly, these scenarios have a *dependency*
> on what is on the local copy of the website, but they are actually
> testing the Tails software (the "Report an Error" launcher for example).

Correct so far.

> They are not testing the content of the local copy of the website itself.

Actually, they *are* testing the content of the local copy, e.g. that
the support page is properly localized. Hence there could be a subset of
automated tests that would make sense to run for doc/ and web/ branches.

Cheers!

___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[sajolida]

anonym:
> [Moving discussion to tails-dev@]

Meta: I really don't want to understand everything that's in this email
but I felt you would want me to answer this one. But if you think that
you can have this discussion without me I would be super happy as well.

> Given the trimming that has happened, some context may have been lost.
> The discussion is about that we now, in our Jenkins setup, automatically
> test images built from doc/ and web/ branches, which wastes a lot of
> time on our isotesters.
> 
>> From: intrigeri 
>> Date: Tue, 20 Oct 2015 13:08:31 +0200
>>
>>> From: bertagaz 
>>> Date: Mon, 19 Oct 2015 12:29:26 +0200
>>
 From: anonym 
>>>
 Still, once we release 1.7, then all doc/ and web/ branches will become
 tested. I suspect we will need a permanent fix for only building (not
 testing) these branches -- it's useless to test them 99.9% of the time,
 and they will block (for ~5 hours) test runs from relevant branches that
 got something pushed to them right after them.
>>
>>> That's something we didn't decide when during the design round. Sounds
>>> doable, but I wonder if there are still some valid points to still test
>>> that branches.
> 
> True, but there's an overwhelming amount of them, and their
> modifications are limited to something that is completely isolated from
> most of Tails, the OS, meaning that a huge part of each test run on them
> is just a (possibly out-dated) re-run of master/devel/stable depending
> on which branch it was based on. That is unlike feature/, bugfix/ and
> test/ branches, that need a full run in general. Perhaps you can see
> where I'm going:
> 
> As an optimization, we could introduce @web and @doc tags, and run the
> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
> respectively. Then we could even have automated tests of @web changes
> before deploying them by browsing the local wiki in Tails. :)
> 
> Note that I may not have the correct understanding of the doc/ vs web/
> distinction, so I'd like a clarification just in case so we don't design
> something stupid. I suspect that since we don't have any automated tests
> for the *website* (as opposed to the docs) we only care about doc/ and
> only need to test web/ if we want to start testing the website.
> 
>> FTR I dislike the idea of blacklisting such branches based on their
>> name. I'm not going to debate it here [...]

The prefixes doc/ and web/ are used loosely to differentiate work on the
"documentation" (/doc /support) and the "website" in general (structure,
stuff not in /doc, etc.) but the difference is not strict.

I also don't think they should be tested. Maybe you could exclude them
from testing by their diff to their base branch. If all the diff is
under wiki/src then don't test that branch.

> Please do it here, then! :) I guess it's related to that you want at
> least doc/ branches to be tested, since our automated test suite
> actually depend on the documentation (see two quotes below for more on
> this).
> 

[...] Skipping tons of stuff I don't understand.

>> [...] making sure that the workaround is not worst than the problem
>> it fixes
> 
> The only effect should be that we won't get automated tests of the few
> scenarios that looks at the wiki shipped inside Tails. I think that
> currently is one scenario: The Report an Error launcher will open the
> support documentation in supported non-English locales. So, if some wiki
> change makes the 'the support documentation page opens in Tor Browser'
> step fail. The only way that can happen is if:
> 
> * the url to the "Help and Support" section changes so the desktop
> shortcut is wrong
> 
> * there's a change to the "Help and Support" section, or its German
> translation
> 
> * we change the website layout of the "parentlinks" thing, or the "home"
> icon there
> 
> Only in the first case would our test suite find an actual error, the
> other two just implies that our test is now out of sync and must be
> updated. Clearly it's not worth wasting that much isotester time on this
> at the moment.

Agreed. If I understand correctly, these scenarios have a *dependency*
on what is on the local copy of the website, but they are actually
testing the Tails software (the "Report an Error" launcher for example).
They are not testing the content of the local copy of the website itself.
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Releasing automated tests

[anonym]

[Moving discussion to tails-dev@]

Given the trimming that has happened, some context may have been lost.
The discussion is about that we now, in our Jenkins setup, automatically
test images built from doc/ and web/ branches, which wastes a lot of
time on our isotesters.

> From: intrigeri 
> Date: Tue, 20 Oct 2015 13:08:31 +0200
> 
>> From: bertagaz 
>> Date: Mon, 19 Oct 2015 12:29:26 +0200
> 
>>> From: anonym 
>> 
>>> Still, once we release 1.7, then all doc/ and web/ branches will become
>>> tested. I suspect we will need a permanent fix for only building (not
>>> testing) these branches -- it's useless to test them 99.9% of the time,
>>> and they will block (for ~5 hours) test runs from relevant branches that
>>> got something pushed to them right after them.
> 
>> That's something we didn't decide when during the design round. Sounds
>> doable, but I wonder if there are still some valid points to still test
>> that branches.

True, but there's an overwhelming amount of them, and their
modifications are limited to something that is completely isolated from
most of Tails, the OS, meaning that a huge part of each test run on them
is just a (possibly out-dated) re-run of master/devel/stable depending
on which branch it was based on. That is unlike feature/, bugfix/ and
test/ branches, that need a full run in general. Perhaps you can see
where I'm going:

As an optimization, we could introduce @web and @doc tags, and run the
test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches,
respectively. Then we could even have automated tests of @web changes
before deploying them by browsing the local wiki in Tails. :)

Note that I may not have the correct understanding of the doc/ vs web/
distinction, so I'd like a clarification just in case so we don't design
something stupid. I suspect that since we don't have any automated tests
for the *website* (as opposed to the docs) we only care about doc/ and
only need to test web/ if we want to start testing the website.

> FTR I dislike the idea of blacklisting such branches based on their
> name. I'm not going to debate it here [...]

Please do it here, then! :) I guess it's related to that you want at
least doc/ branches to be tested, since our automated test suite
actually depend on the documentation (see two quotes below for more on
this).

I wonder if we could have a configuration filem like
config/jenkins_testing, which has settings controlling how the automated
test suite is invoked on Jenkins. I imagine something like this:

ENABLE = {true,false}

Whether to run the test suite at all. Hence, when merging another base
branch into master as part of the release process, we just make sure to
set this to `false` and we're done. When merging master into a base
branch, it must be set to `true`. So it's similar to config/base_branch,
except that it includes master as a special case. Is this problematic?
Won't merge conflicts save us?


If we prefer it, we could implement my tagging idea with this:

CUCUMBER_ARGUMENTS = 

This makes Jenkins invoke run_test_suite with extra CUCUMBER_ARGUMENTS.
And it could be used for the @web/@doc idea above if we have it set to
`--tag @doc` or whatever in master. We could even test master then, in
the same way. We just have to update the release process in the same way
as for the ENABLE setting, but for CUCUMBER_ARGUMENTS.

Since this affects how Jenkins will invoke a script, there's a security
concern vs the shell that has to be considered. Also arbitrary paths can
be given. Hence we probably may want to have this option *instead*:

RUN_WITH_TAGS = 

So if it's set to ["tag1", "~tag2,tag3"] it would be translated to these
distinct arguments:

['--tag', 'tag1', '--tag', '~tag2,tag3']

in the usual safe (i.e. shell-free) list syntax used for exec in many
programming languages (python, ruby, perl...). This would also improve
the situation for @fragile, and also be subject to the same release
process update as ENABLE above.



Note that ENABLE gives an alternative implementation for the problem
that #10389 wants to solve, i.e. "don't run tests for WIP branches". I
think I prefer this approach, so we avoid excessive branch re-naming.

Also note that with the flexibility from
CUCUMBER_ARGUMENTS/CUCUMBER_TAGS we can remove `--tag ~@fragile` from
the Jenkins wrapper script (#10288) by setting it in the config instead.

Using CUCUMBER_ARGUMENTS/CUCUMBER_TAGS the true power users could even
do "focus" testing of specific features/scenarios via a temporary tag
and then making Jenkins only test that tag -- now the test suite will
not waste time on other scenarios. FWIW. I don't think this will play
nicely with our future ideas about Jenkins updating tickets in Redmine
given the test results. That could be fixed if we added another option:

JENKINS_REVIEW = {true,false}

It would be true by default (I guess), and would then enable Jenkins