Bug#834682: mina2: FTBFS in testing (failing tests)

[tony mancill]

On Fri, Sep 16, 2016 at 07:35:58PM +0200, Santiago Vila wrote:
> Hello.
> 
> I tried building this package 20 times this morning.
> 
> It failed 19 times. The build logs are attached in a single tarball.
> 
> There is something which may help in this bug: there is a new upstream
> release of mina available (2.0.14).
> 
> So I'd like to propose that you or anybody in the team maintaining
> this package do this:
> 
> * Raise this to serious (where it belongs, it's a FTBFS bug).
> 
> * Disable completely the tests for this version (trivial patch attached).
>   The upload doing that would naturally close this bug.
> 
>   Disabling only the tests that fail would be also an option, but the tests
>   failing for me are not the same failing for Gregor, and it would not be
>   really productive or useful to determine which of the tests are ok and
>   which ones are not when we have a new upstream release available that may
>   have some or all of the failing tests already fixed.
> 
> * Try to upload mina version 2.0.14 for unstable (but this time without the 
> patch)
>   to see if things improve with the new version. Hopefully this version does 
> not
>   fail anymore, or maybe it fails for you and for me at the same time.
> 
> Would this plan be acceptable to you?
> 
> If yes, please go ahead and stop reading. Thanks a lot!
> 
> If not: there is a tag called stretch-ignore which is the right way to
> make a serious bug like this one not to be RC:
> 
> https://release.debian.org/stretch/rc_policy.txt
> 
> Please read the relevant paragraph: You would have to ask the Release
> Managers for permission first, you can't just downgrade a FTBFS bug on
> your own as a way to make it not RC.
> 
> But I really believe we can skip all that. Believe me, it is not my
> intention or desire to see this package autoremoved from Debian or
> anything like that, everything I ask is that it builds ok, and that's
> very easy to achieve with the attached path.
> 
> Thanks.

Hi Santiago,

Ugh - thank you for putting so much effort into this and for providing
logs.  I have tried to get the build to fail for me locally and can't,
which is also frustrating.  However, I feel like we have an established
precedent for disabling non-deterministic test suites, since flaky tests
can be worse than no tests at all, so I am applying your patch and
preparing an upload.

We can always enable tests down the road, as you suggest, perhaps with
the upload of the next upstream version.  

Cheers,
tony


signature.asc
Description: PGP signature
__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#834682: mina2: FTBFS in testing (failing tests)

[Santiago Vila]

On Thu, 8 Sep 2016, gregor herrmann wrote:

> Logs attached. Unfortunately I don't have any idea on how to fix the
> problem, but if you want me to do something else just shout.

Well, if you don't mind burning a bunch of additional CPU cycles,
I would continue to build it several times until it fails again.

(If you tried three times and it failed once, it seems likely that it
happens again).

Then it would be interesting to see if the test which fails is the
same it failed the first time, which, in your case, was this one:

Failed tests: 
  DatagramBindTest>AbstractBindTest.testUnbindDisconnectsClients:187
expected:<5> but was:<4>

I have not analyzed my build logs, but I guess this is unfortunately
not the same test which fails for me.


BTW: I try to be objective by never reporting something when it only
happens to me. This is why I sometimes say "here is a build log and it
also fails in reproducible builds" or also "here you have several
build logs from several different autobuilders".

I think it would be just fair to ask the maintainer to be objective as
well and consider whether it is really justified to downgrade the
severity of a FTBFS bug when everybody else get a FTBFS too and his
computer is the only one in which it may not be reproduced.

If it is certainly frustrating when one receives a bug which is random
and difficult to reproduce, imagine the frustration of the bug
reporter who sees his report being downgraded or even closed (!)
(see Bug #834964 for a very recent example) when it's quite clear
that the package does not really "build from source".

Thanks a lot.

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#834682: mina2: FTBFS in testing (failing tests)

[Santiago Vila]

On Wed, 7 Sep 2016, Markus Koschany wrote:

> I am attaching two build logs (cowbuilder and sbuild) that show no
> issues at all. The build succeeds.

Thanks a lot for trying to reproduce this.

There are still a lot of differences, we should better remove them all,
until our building environments are the same. Then it should fail
for you too (hopefully).

Would you please add this to the mix?:

* Use a stretch chroot (as reported).

* Build only arch-independent packages (as reported).
  This is the equivalent to "dpkg-buildpackage -A".
  In sbuild it's done with options "--arch-all --no-arch-any".

* The chroot should be as small as possible. Well, I don't do exactly
  that because most packages build-depend on debhelper, and I want
  to save disk I/O so I have debhelper by default, but on the other side
  I don't have gnupg installed.
  
  To simplify: Would you please install the packages in
  package-list.txt (attached) and only those?
  
[ This is of course only the starting point, sbuild should then
  automatically download and install the required packages ].

* I'm also using --resolve-alternatives option of sbuild. Don't know if
  it's relevant but we should try to minimize differences. There is at
  least one package that FTBFS if I didn't use this option, which is
  otherwise the default (AFAIK) in the official buildds.

* Build with only one CPU. This is achieved by putting this in .sbuildrc:

$ENV{'DEB_BUILD_OPTIONS'} = 'parallel=1';

* Finally, I'm building on machines with not a lot of memory, because
  monitoring /proc/meminfo tells me that I don't need more for this
  particular package (but maybe I'm wrong).
  
  Please try on a virtual machine having only 1GB RAM and 1 GB swap.


If you do all this, the probability that you will reproduce the bug
may only increase (but of course it may also continue to be zero).
In either case, we should try.

Thanks.adduser
apt
autoconf
automake
autopoint
autotools-dev
base-files
base-passwd
bash
binutils
bsdmainutils
bsdutils
build-essential
bzip2
coreutils
cpp
cpp-6
dash
debconf
debhelper
debian-archive-keyring
debianutils
dh-autoreconf
dh-strip-nondeterminism
diffutils
dmsetup
dpkg
dpkg-dev
e2fslibs
e2fsprogs
eatmydata
fakeroot
file
findutils
g++
g++-6
gcc
gcc-5-base
gcc-6
gcc-6-base
gettext
gettext-base
gpgv
grep
groff-base
gzip
hostname
init-system-helpers
initscripts
insserv
intltool-debian
less
libacl1
libapt-pkg5.0
libarchive-zip-perl
libasan3
libatomic1
libattr1
libaudit-common
libaudit1
libblkid1
libbsd0
libbz2-1.0
libc-bin
libc-dev-bin
libc-l10n
libc6
libc6-dev
libcap-ng0
libcc1-0
libcilkrts5
libcomerr2
libcroco3
libdb5.3
libdebconfclient0
libdevmapper1.02.1
libdpkg-perl
libeatmydata1
libfakeroot
libfdisk1
libffi6
libfile-stripnondeterminism-perl
libgcc-6-dev
libgcc1
libgcrypt20
libgdbm3
libglib2.0-0
libgmp10
libgomp1
libgpg-error0
libicu57
libisl15
libitm1
liblsan0
liblz4-1
liblzma5
libmagic-mgc
libmagic1
libmount1
libmpc3
libmpfr4
libmpx2
libncurses5
libncursesw5
libpam-modules
libpam-modules-bin
libpam-runtime
libpam0g
libpcre3
libperl5.22
libpipeline1
libquadmath0
libselinux1
libsemanage-common
libsemanage1
libsepol1
libsigsegv2
libsmartcols1
libss2
libstdc++-6-dev
libstdc++6
libsystemd0
libtimedate-perl
libtinfo5
libtool
libtsan0
libubsan0
libudev1
libunistring0
libustr-1.0-1
libuuid1
libxml2
linux-libc-dev
locales
login
lsb-base
m4
make
man-db
mawk
mount
multiarch-support
nano
ncurses-base
ncurses-bin
passwd
patch
perl
perl-base
perl-modules-5.22
po-debconf
sed
sensible-utils
startpar
sysv-rc
sysvinit-utils
tar
tzdata
util-linux
xz-utils
zlib1g
__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#834682: mina2: FTBFS in testing (failing tests)

[Markus Koschany]

On 07.09.2016 14:33, Santiago Vila wrote:
> retitle 834682 mina2: FTBFS in testing (failing tests)
> thanks
> 
> On Fri, 26 Aug 2016, Markus Koschany wrote:
> 
>> I have rebuilt mina2 ten times in a row now but I can't reproduce the
>> failing test or the build failure in general.
> 
> Congratulations. I attach five build logs in five different autobuilders.
> 
> Apparently, it fails for me all the time.
> 
> [ So, instead of you asking me for a way to reproduce it, perhaps I
>   should be asking you for a way *not* to reproduce it... ]
> 
>> It appears this issue is not random
> 
> After my last five builds yesterday, that's what it seems, yes.
> Now it fails all the time, so it's not random anymore.
> 
> I'm changing the subject accordingly.

I am attaching two build logs (cowbuilder and sbuild) that show no
issues at all. The build succeeds.

I have fixed previous packages with random failing tests because I think
those tests are not helpful for us in general but I won't disable them
if I can't reproduce the randomness of the issue myself. I always try to
reproduce bug reports but I am getting more and more frustrated by the
way how you interact with us. It is just annoying and disrespectful.



mina2_2.0.13-1_amd64.build.gz
Description: application/gzip


mina2_2.0.13-1_amd64-2016-09-07T12:40:38Z.build.gz
Description: application/gzip


signature.asc
Description: OpenPGP digital signature
__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.