On Tue, Feb 5, 2013, at 06:21 PM, Greg Banks wrote:
>
>
> On Mon, Feb 4, 2013, at 09:12 PM, Jenkins wrote:
> > See <http://ci.cyrusimap.org/job/cyrus-imapd-master/1126/>
> >
> > Test failures and errors summary
> > ================================
> > [...]
> >
> > 191 run, 0 failures, 33 errors
>
> Every single one of these was a spurious failure due to poor isolation
> of bound ports, like this
>
> Perl exception: Some process is already listening on 127.0.0.1:9107
> at Cassandane/Instance.pm line 490
>
> Cassandane::Instance::_start_master('Cassandane::Instance=HASH(0x1bf6c7d0)')
> called at Cassandane/Instance.pm line 708
> Cassandane::Instance::start('Cassandane::Instance=HASH(0x1bf6c7d0)')
> called at Cassandane/Cyrus/TestCase.pm line 187
>
> Cassandane::Cyrus::TestCase::_start_instances('Cassandane::Cyrus::Sieve=HASH(0x1b8194d0)')
> called at Cassandane/Cyrus
>
> I have some commits on another branch which improve test isolation by
> using netstat -p to discover processes which are bound to the test ports
> and still running after each test finishes, and killing them off. I'll
> get those pushed out sometime tomorrow.
Well, that took longer than expected, but it turned out that running
Cassandane with -j8 a lot exposed some pre-existing imapd bugs that I
needed to fix first. Anyway I've just pushed these commits to
Cassandane.
b061d68 Add wrapper for dovecot's ImapTest suite
8c53923 Compress Berkeley environment files on stop
3b29efe Allow specifying the message size in bytes
8c15a2d Allow date to be specified as an ISO8601 string
d03ceb3 Work around old Config::IniFiles
4349c78 Ensure core files are readable
fa3124d Dump log to stderr if > one -v option given
8cf76c6 Add support for parameterised tests
468fbd9 Kill stray processes on test exit
a1de6a6 Log files are world readable
4ed8fd9 Cleanup start-instance.pl
dde24b5 Parse cyrus.conf of pre-existing on-disk instances
a8e4042 Add class methods list() and exists() to Instance
2aa8b21 Add Cassandane::Daemon::parse_address()
3f6eb04 Introduce class Mboxname
325546a Add --log-directory option to testrunner
d5d51b7 Use SIGTERM for graceful shutdown not SIGQUIT
f3a08ae Unified xlog() and process stderr/stdout
b2ac40e Whitespace fix
bf6668b Replace config_TEST methods with magic attributes
9dd7d7c Add --latest argument to start-instance.pl
3472d86 Update default parameters for genmail script et al
851f2d9 Create a default boundary for multipart messages
c6916d6 Generator allows overriding MIME parameters
de9bde4 run_replication now takes many options
b9ed278 stop_command works even if command needs reaping
381b514 Add support for daemons not managed by master
260827d Idle: add a test for shutting down Cyrus with IDLE
c674e84 Allow setting Cc: and Bcc: in Generator
9faea63 make_random_address() is now public and has params
Most of these make the job of running Cassandane manually easier, or
clean up the Cassandane code a littlle, but some deserve special
mention.
9dd7d7c Add --latest argument to start-instance.pl
The start-instance.pl script will manually start a Cyrus instance given
it's name. The --latest option allows you to start the most recently
run instance, e.g. the one from the failing Cassandane test you just
ran, for manual debugging. I also fixed a bug where start-instance.pl
didn't work on Cyrus instances created when Cassandane is run in
parallel mode.
325546a Add --log-directory option to testrunner
etc
The logging behaviour of testrunner is now different. Each test's
stdout, stderr, Cassandane logs, and syslogs (for some Cyrus processes
only) are written together to a per-test file as the test runs. If the
test fails, this log file is shown as "Annotations" in the testrunner
report. If the test succeeds, the log is removed. If you want always
to keep the logs, the --log-directory/-L option specifies a directory
into which the logs are written (and not removed).
468fbd9 Kill stray processes on test exit
This commit has, in my experience, fixed the problem of leaked processes
holding down test ports and causing spurious cascading failures. Here's
hoping it works on ci.cyrusimap.org, which is a RH-like environment (and
has subtle differences in the output of netstat).
8cf76c6 Add support for parameterised tests
I'm quite pleased about this one. It allows a single test function to
be run automatically multiple times with combinations of parameters.
For example
my $mustache;
Cassandane::Unit::TestCase::parameter(\$mustache, 'walrus',
'toothbrush', 'waxed');
my $nose;
Cassandane::Unit::TestCase::parameter(\$nose, 'roman');
my $eyes;
Cassandane::Unit::TestCase::parameter(\$eyes, 'brown', 'cat');
sub test_face
{
xlog "XXX face: mustache=$mustache eyes=$eyes nose=$nose";
}
Results in
% head -2 logs/Cassandane.Test.Parameter.face.*
==> logs/Cassandane.Test.Parameter.face.3Veyoi.log <==
=====> Test::Parameter[66] XXX face: mustache=waxed eyes=cat nose=roman
==> logs/Cassandane.Test.Parameter.face.DKKHYE.log <==
=====> Test::Parameter[66] XXX face: mustache=toothbrush eyes=cat
nose=roman
==> logs/Cassandane.Test.Parameter.face.fBOIpP.log <==
=====> Test::Parameter[66] XXX face: mustache=toothbrush eyes=brown
nose=roman
==> logs/Cassandane.Test.Parameter.face.fH0isI.log <==
=====> Test::Parameter[66] XXX face: mustache=walrus eyes=brown
nose=roman
==> logs/Cassandane.Test.Parameter.face.HOMsdd.log <==
=====> Test::Parameter[66] XXX face: mustache=waxed eyes=brown
nose=roman
==> logs/Cassandane.Test.Parameter.face.PCstBb.log <==
=====> Test::Parameter[66] XXX face: mustache=walrus eyes=cat nose=roman
I expect to use this soon to expand the coverage of tests with respect
to various Cyrus parameters like delete_mode and unixhierarchysep.
b061d68 Add wrapper for dovecot's ImapTest suite
I've been using this internally in fastmail to add Dovecot's system
tests to Cassandane's instance management, parallelism, and reporting,
thus expanding our test coverage. The downside is that you need to
download and build Dovecot to get some libraries, which makes it
somewhat awkward to set up on ci.cyrusimap.org.
--
Greg.
