On Mon, Jan 7, 2013, at 09:13 AM, Jenkins wrote:
> See <http://ci.cyrusimap.org/job/cyrus-imapd-master/1069/>
>
> ------------------------------------------
> [...truncated 4455 lines...]
> + '[' -d .git ']'
> ++ git ls-files
> ++ wc -l
> + nfiles=99
> + '[' 99 -gt 0 ']'
> + git ls-files -o
> broken.log
> cass.errs
> cassandane.ini
> coverage.xml
> e.fpl
> errs
> errs2
> find-process-leak.sh
> reports.old/TEST-Cassandane.Cyrus.Annotator.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3072.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3463.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3470.xml
> reports.old/TEST-Cassandane.Cyrus.Bug3649.xml
> reports.old/TEST-Cassandane.Cyrus.Delete.xml
> reports.old/TEST-Cassandane.Cyrus.Delivery.xml
> reports.old/TEST-Cassandane.Cyrus.Fetch.xml
> reports.old/TEST-Cassandane.Cyrus.Flags.xml
> reports.old/TEST-Cassandane.Cyrus.Idle.xml
> reports.old/TEST-Cassandane.Cyrus.Info.xml
> reports.old/TEST-Cassandane.Cyrus.Lsub.xml
> reports.old/TEST-Cassandane.Cyrus.Master.xml
> reports.old/TEST-Cassandane.Cyrus.Metadata.xml
> reports.old/TEST-Cassandane.Cyrus.Nntp.xml
> reports.old/TEST-Cassandane.Cyrus.Pop3.xml
> reports.old/TEST-Cassandane.Cyrus.Quota.xml
> reports.old/TEST-Cassandane.Cyrus.Rename.xml
> reports.old/TEST-Cassandane.Cyrus.Replication.xml
> reports.old/TEST-Cassandane.Cyrus.Search.xml
> reports.old/TEST-Cassandane.Cyrus.Sieve.xml
> reports.old/TEST-Cassandane.Cyrus.Simple.xml
> reports.old/TEST-Cassandane.Test.Address.xml
> reports.old/TEST-Cassandane.Test.Cassini.xml
> reports.old/TEST-Cassandane.Test.Clone.xml
> reports.old/TEST-Cassandane.Test.Config.xml
> reports.old/TEST-Cassandane.Test.DateTime.xml
> reports.old/TEST-Cassandane.Test.Message.xml
> reports.old/TEST-Cassandane.Test.MessageStoreFactory.xml
> reports.old/TEST-Cassandane.Test.Metronome.xml
> reports.old/TEST-Cassandane.Test.Sample.xml
> reports/TEST-Cassandane.Cyrus.Annotator.xml
> reports/TEST-Cassandane.Cyrus.Bug3072.xml
> reports/TEST-Cassandane.Cyrus.Bug3463.xml
> reports/TEST-Cassandane.Cyrus.Bug3470.xml
> reports/TEST-Cassandane.Cyrus.Bug3649.xml
> reports/TEST-Cassandane.Cyrus.Delete.xml
> reports/TEST-Cassandane.Cyrus.Delivery.xml
> reports/TEST-Cassandane.Cyrus.Fetch.xml
> reports/TEST-Cassandane.Cyrus.Flags.xml
> reports/TEST-Cassandane.Cyrus.Idle.xml
> reports/TEST-Cassandane.Cyrus.Info.xml
> reports/TEST-Cassandane.Cyrus.Lsub.xml
> reports/TEST-Cassandane.Cyrus.Master.xml
> reports/TEST-Cassandane.Cyrus.Metadata.xml
> reports/TEST-Cassandane.Cyrus.Nntp.xml
> reports/TEST-Cassandane.Cyrus.Pop3.xml
> reports/TEST-Cassandane.Cyrus.Quota.xml
> reports/TEST-Cassandane.Cyrus.Rename.xml
> reports/TEST-Cassandane.Cyrus.Replication.xml
> reports/TEST-Cassandane.Cyrus.Search.xml
> reports/TEST-Cassandane.Cyrus.Sieve.xml
> reports/TEST-Cassandane.Cyrus.Simple.xml
> reports/TEST-Cassandane.Test.Address.xml
> reports/TEST-Cassandane.Test.Cassini.xml
> reports/TEST-Cassandane.Test.Clone.xml
> reports/TEST-Cassandane.Test.Config.xml
> reports/TEST-Cassandane.Test.DateTime.xml
> reports/TEST-Cassandane.Test.Message.xml
> reports/TEST-Cassandane.Test.MessageStoreFactory.xml
> reports/TEST-Cassandane.Test.Metronome.xml
> reports/TEST-Cassandane.Test.Sample.xml
> utils/gdbtramp
> utils/gdbtramp.o
> utils/lemming
> utils/lemming.o
> working.log
> + git status
> # On branch master
> # Untracked files:
> # (use "git add <file>..." to include in what will be committed)
> #
> # broken.log
> # coverage.xml
> # e.fpl
> # errs
> # errs2
> # find-process-leak.sh
> # working.log
> nothing added to commit but untracked files present (use "git add" to
> track)
> + make
> make[1]: Entering directory
> `<http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/utils'>
> make[1]: Nothing to be done for `all'.
> make[1]: Leaving directory
> `<http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/cassandane/utils'>
> testrunner.pl syntax OK
> Cassandane/ThreadedGenerator.pm syntax OK
> Cassandane/Config.pm syntax OK
> Cassandane/MasterEntry.pm syntax OK
> Cassandane/Cyrus/Quota.pm syntax OK
> Cassandane/Cyrus/Delivery.pm syntax OK
> Cassandane/Cyrus/Conversations.pm syntax OK
> Cassandane/Cyrus/Rename.pm syntax OK
> Cassandane/Cyrus/Bug3463.pm syntax OK
> Cassandane/Cyrus/Fetch.pm syntax OK
> Cassandane/Cyrus/Bug3470.pm syntax OK
> Cassandane/Cyrus/Delete.pm syntax OK
> Cassandane/Cyrus/Pop3.pm syntax OK
> Cassandane/Cyrus/Metadata.pm syntax OK
> Cassandane/Cyrus/Info.pm syntax OK
> Cassandane/Cyrus/Nntp.pm syntax OK
> Cassandane/Cyrus/Bug3649.pm syntax OK
> Cassandane/Cyrus/TestCase.pm syntax OK
> Cassandane/Cyrus/Sieve.pm syntax OK
> Cassandane/Cyrus/Replication.pm syntax OK
> Cassandane/Cyrus/Master.pm syntax OK
> Cassandane/Cyrus/Flags.pm syntax OK
> Cassandane/Cyrus/Lsub.pm syntax OK
> Cassandane/Cyrus/Idle.pm syntax OK
> Cassandane/Cyrus/Bug3072.pm syntax OK
> Cassandane/Cyrus/Search.pm syntax OK
> Cassandane/Cyrus/Annotator.pm syntax OK
> Cassandane/Cyrus/Simple.pm syntax OK
> Cassandane/Test/Config.pm syntax OK
> Cassandane/Test/Sample.pm syntax OK
> Cassandane/Test/Message.pm syntax OK
> Cassandane/Test/DateTime.pm syntax OK
> Cassandane/Test/MessageStoreFactory.pm syntax OK
> Cassandane/Test/Metronome.pm syntax OK
> Cassandane/Test/Clone.pm syntax OK
> Cassandane/Test/Address.pm syntax OK
> Cassandane/Test/Cassini.pm syntax OK
> Cassandane/Instance.pm syntax OK
> Cassandane/Message.pm syntax OK
> Cassandane/POP3MessageStore.pm syntax OK
> Cassandane/SequenceGenerator.pm syntax OK
> Cassandane/PortManager.pm syntax OK
> Cassandane/IMAPService.pm syntax OK
> Cassandane/MessageStoreFactory.pm syntax OK
> Cassandane/IMAPMessageStore.pm syntax OK
> Cassandane/MaildirMessageStore.pm syntax OK
> Cassandane/Unit/SyslogListener.pm syntax OK
> Cassandane/Unit/RunnerPretty.pm syntax OK
> Cassandane/Unit/RunnerXML.pm syntax OK
> Cassandane/Unit/TestCase.pm syntax OK
> Cassandane/Unit/TestPlan.pm syntax OK
> Cassandane/Unit/Runner.pm syntax OK
> Cassandane/MessageStore.pm syntax OK
> Cassandane/Generator.pm syntax OK
> Cassandane/ServiceFactory.pm syntax OK
> Cassandane/Address.pm syntax OK
> Cassandane/MboxMessageStore.pm syntax OK
> Cassandane/Service.pm syntax OK
> Cassandane/Util/SHA.pm syntax OK
> Cassandane/Util/Sample.pm syntax OK
> Cassandane/Util/Wait.pm syntax OK
> Cassandane/Util/DateTime.pm syntax OK
> Cassandane/Util/Words.pm syntax OK
> Cassandane/Util/Setup.pm syntax OK
> Cassandane/Util/Metronome.pm syntax OK
> Cassandane/Util/Log.pm syntax OK
> Cassandane/Util/NetString.pm syntax OK
> Cassandane/Util/Socket.pm syntax OK
> Cassandane/MasterEvent.pm syntax OK
> Cassandane/Cassini.pm syntax OK
> Cassandane/MasterStart.pm syntax OK
> + sed -e 's|^##destdir =.*$|destdir =
> <http://ci.cyrusimap.org/job/cyrus-imapd-master/ws/inst|'> -e
> 's|^##pwcheck = .*$|pwcheck = sasldb|'
> + rm -rf reports.old
> + mv -f reports reports.old
> + mkdir -m 0777 reports
> + ./testrunner.pl --cleanup -f xml -v
> + exitcode=1
> + '[' -x jenkins-xml-summary.pl ']'
> + ./jenkins-xml-summary.pl
> --build-url=http://ci.cyrusimap.org/job/cyrus-imapd-master/1069/
> Test failures and errors summary
> ================================
>
> Cassandane::Cyrus::Nntp.cve_2011_3208_newnews
>
> http://ci.cyrusimap.org/job/cyrus-imapd-master/1069//testReport/%28root%29/Cassandane__Cyrus__Nntp/test_cve_2011_3208_newnews/
Error Message
Perl exception: Cannot shut down master pid 28883
Stacktrace
test_cve_2011_3208_newnews(Cassandane::Cyrus::Nntp)
Perl exception: Cannot shut down master pid 28883
at Cassandane/Instance.pm line 831
Cassandane::Instance::stop('Cassandane::Instance=HASH(0x18d313c0)')
called at Cassandane/Cyrus/TestCase.pm line 267
Cassandane::Cyrus::TestCase::tear_down('Cassandane::Cyrus::Nntp=HASH(0x18490550)')
called at Cassandane/Cyrus/Nntp.pm line 83
Cassandane::Cyrus::Nntp::tear_down('Cassandane::Cyrus::Nntp=HASH(0x18490550)')
called at /usr/lib/perl5/vendor_perl/5.8.8/Test/Unit/TestCase.pm
line 65
What happened here is most likely:
* the Cassandane Perl code initiated shutdown by sending SIGTERM to the
master process
* the Perl code waits, for up to 20 sec
* master resent the signal to it's process group
* master waits (with no timeout) to be notified of the graceful exit of
its children via SIGCHLD
* but a child (in this case, probably the imapd that was used to issue
admin commands to set up the instance at the start of the test) failed
to notice the SIGTERM and never died
* eventually the Perl code wakes up and notices that the master process
has not exited
* the Perl code wreaks terrible vengeance with SIGILL or SIGKILL and
reports the test failed.
I've recently fixed several Cyrus bugs which caused this class of
failure mode, commits
59b181c Also use signals_select() waiting for connection
10c21a1 Also use signals_select() in prot_*() functions
76f5e8e idle: close SIGTERM/select race
The rest of these were spurious cascading failures like
Error Message
Perl exception: Some process is already listening on 127.0.0.1:9107
which should be solved by Cassandane commit
468fbd9 Kill stray processes on test exit
--
Greg.
