Re: [Openvpn-devel] rfd: 'serial-tests' automake option

[Gert Doering]

Hi,

On Wed, May 27, 2015 at 08:41:25PM +0200, Gert Doering wrote:
> On Wed, May 27, 2015 at 02:19:38PM +0200, David Sommerseth wrote:
> > This makes a lot of sense to me.  Our project doesn't really benefit in
> > general of running our tests parallel.
> > 
> > This gets an ACK from me.  I just tested your change locally, and it
> > seems to work very well.  
> 
> Thanks :-)  (and of *course* I tested this before posting...  sometimes I
> actually remember to run "make test" before pushing out broken commits...
> *ahem*)

Unfortunately, this was just *too* easy - in other words, if you happen
to build with an older version of automake that does not understand the
"serial-tests" option, it will just blow up...

autoreconf: running: automake --add-missing --copy --no-force
tests/Makefile.am:12: option `serial-tests' not recognized
autoreconf: automake failed with exit status: 1

... haven't seen *that* much "red" in the buildbot status in quite a while
- basically, everything older than a few years (FreeBSD 8, CentOS 6,
Debian 7, Ubuntu 12.04...)

David suggested making this conditional, but I couldn't find anything in
the docs, so I think I'll revert the patch tomorrow-ish and reopen the 
trac ticket... :-(

(Buildbots can be fixed, of course, but what we have now would blow up in 
the same way for all users of a git checkout.  Thankyouverymuch, dear 
Autotools community...)

gert
-- 
USENET is *not* the non-clickable part of WWW!
   //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025g...@net.informatik.tu-muenchen.de


pgp5_r1wdhNT_.pgp
Description: PGP signature

Re: [Openvpn-devel] rfd: 'serial-tests' automake option

[Gert Doering]

Hi,

On Wed, May 27, 2015 at 02:19:38PM +0200, David Sommerseth wrote:
> This makes a lot of sense to me.  Our project doesn't really benefit in
> general of running our tests parallel.
> 
> This gets an ACK from me.  I just tested your change locally, and it
> seems to work very well.  

Thanks :-)  (and of *course* I tested this before posting...  sometimes I
actually remember to run "make test" before pushing out broken commits...
*ahem*)

> And from what I can grasp from the automake
> documentation [1], this seems safe for us to do.
> 
> [1]
> 

Thanks for this pointer - while the server seems to be down right now, I've
included the URL in the commit.

commit fc03ca9d13e35c40bdf1c3c676db2adf48c60223 (master)
commit c196ea922755bb25f9837080d562ef4d3495999f (release/2.3)

Author: Gert Doering 
List-Post: openvpn-devel@lists.sourceforge.net
Date:   Wed May 27 20:31:38 2015 +0200

Enforce "serial-tests" behaviour for tests/Makefile

Signed-off-by: Gert Doering 
Message-ID: <20150524193011.gk...@greenie.muc.de>
URL: http://article.gmane.org/gmane.network.openvpn.devel/9738
Acked-by: David Sommerseth 



-- 
USENET is *not* the non-clickable part of WWW!
   //www.muc.de/~gert/
Gert Doering - Munich, Germany g...@greenie.muc.de
fax: +49-89-35655025g...@net.informatik.tu-muenchen.de


pgp9SIk8tEq72.pgp
Description: PGP signature

Re: [Openvpn-devel] rfd: 'serial-tests' automake option

[David Sommerseth]

On 24/05/15 21:30, Gert Doering wrote:
> Hi,
> 
> somewhere between automake-1.11 and automake-1.14 the default for handling
> "make check" changed from "serial-tests" to "parallel-tests" - which 
> sounds harmless, but has the interesting side effect of hiding the output
> of all (potentially parallel-run) test scripts until the summary at the
> end, and even then, only printing the summary, not the detail output.
> 
> I can see that this looks much more pretty (and colours, wooh!), but
> for my day-to-day regression testing and for analyzing buildbot failures,
> this just adds extra steps to figure out what it is doing right now, and
> where and why it failed this time - especially in the buildbot case, the
> difference is "the reason is right in the mail" vs "I need to login to
> the buildbot machine, go to the right build directory (of many), and 
> grab the log file from there".
> 
> 
> Fortunately, there is an automake option for this, which I'd propose to
> enable... (well hidden inside "info automake-1.14" -> "Options")
> 
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index b7980e0..02fa392 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -9,6 +9,8 @@
>  #  Copyright (C) 2006-2012 Alon Bar-Lev 
>  #
>  
> +AUTOMAKE_OPTIONS = serial-tests
> +
>  MAINTAINERCLEANFILES = \
> $(srcdir)/Makefile.in
>  
> 
> ... which brings back the old behaviour.
> 
> I'm not sure whether it can be passed on an "autoreconf -vif" command
> line, and if yes, how, but I presume that really nobody who bothers to
> run "make check" will be bothered by the normal test output - and especially
> nobody who bothers to set up a t_client.rc test rig in the first place...
> 
> But I'm open for discussion and alternative suggestions on this.
> 

This makes a lot of sense to me.  Our project doesn't really benefit in
general of running our tests parallel.

This gets an ACK from me.  I just tested your change locally, and it
seems to work very well.  And from what I can grasp from the automake
documentation [1], this seems safe for us to do.


[1]


-- 
kind regards,

David Sommerseth




signature.asc
Description: OpenPGP digital signature

Re: [Openvpn-devel] rfd: 'serial-tests' automake option

[Samuli Seppänen]

Hi,

somewhere between automake-1.11 and automake-1.14 the default for handling
"make check" changed from "serial-tests" to "parallel-tests" - which
sounds harmless, but has the interesting side effect of hiding the output
of all (potentially parallel-run) test scripts until the summary at the
end, and even then, only printing the summary, not the detail output.

I can see that this looks much more pretty (and colours, wooh!), but
for my day-to-day regression testing and for analyzing buildbot failures,
this just adds extra steps to figure out what it is doing right now, and
where and why it failed this time - especially in the buildbot case, the
difference is "the reason is right in the mail" vs "I need to login to
the buildbot machine, go to the right build directory (of many), and
grab the log file from there".


Fortunately, there is an automake option for this, which I'd propose to
enable... (well hidden inside "info automake-1.14" -> "Options")


Running the tests in serial and having the failures show up in buildbot 
emails makes sense to me. Consider this a feature-ACK.


--
Samuli Seppänen
Community Manager
OpenVPN Technologies, Inc

irc freenode net: mattock