Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 7/5/24 a las 19:43, Uecker, Martin escribió: Yes, if you do not mind to do the work, please update bullseye. That would be much appreciated. Hi. The uploads for bart and bart-cuda have been accepted for bullseye-proposed-updates. So I pushed the pending changes to salsa. One thing I didn't mention before: I noticed that both packages share the same git repo, so I understand that only the releases of "bart" can have git tags on such repository. Maybe we could use a special tag format for the contrib branches, but I guess that it does not worth the effort. Thanks.
Bug#1026061: [+externe Mail+] Re: Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Am Dienstag, dem 07.05.2024 um 19:23 +0200 schrieb Santiago Vila: > El 7/5/24 a las 18:50, Uecker, Martin escribió: > > Am Dienstag, dem 07.05.2024 um 17:59 +0200 schrieb Santiago Vila: > > > El 1/1/23 a las 16:55, Uecker, Martin escribió: > > > In the meantime, I became member of debian-med, so in theory, > > > I could fix this myself via team upload. Would you prefer > > > that I take care of this myself that way? > > > > I wouldn't mind if you did. There are some other bugs > > which could easily be fixed with a new upload (i.e. with > > minor patches in the bug tracker) > > > > > > (I would also handle bart-cuda, also affected but not reported yet) > > > > bart-cuda needs to be updated to 0.9.00 which should > > be straightforward in principle but need more work > > and a binary upload. > > Well, just in case: I'm talking about making an upload > for bullseye, using the same fix in bookworm, then > sending a bug to release.debian.org explaining the > issue, etc. > > That's mainly bureaucratic and will not interfere with your normal > work in unstable regarding new upstream releases and such (which > I still prefer not to handle myself). > Yes, if you do not mind to do the work, please update bullseye. That would be much appreciated. Martin
Bug#1026061: [+externe Mail+] Re: Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 7/5/24 a las 18:50, Uecker, Martin escribió: Am Dienstag, dem 07.05.2024 um 17:59 +0200 schrieb Santiago Vila: El 1/1/23 a las 16:55, Uecker, Martin escribió: In the meantime, I became member of debian-med, so in theory, I could fix this myself via team upload. Would you prefer that I take care of this myself that way? I wouldn't mind if you did. There are some other bugs which could easily be fixed with a new upload (i.e. with minor patches in the bug tracker) (I would also handle bart-cuda, also affected but not reported yet) bart-cuda needs to be updated to 0.9.00 which should be straightforward in principle but need more work and a binary upload. Well, just in case: I'm talking about making an upload for bullseye, using the same fix in bookworm, then sending a bug to release.debian.org explaining the issue, etc. That's mainly bureaucratic and will not interfere with your normal work in unstable regarding new upstream releases and such (which I still prefer not to handle myself). Thanks.
Bug#1026061: [+externe Mail+] Re: Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Am Dienstag, dem 07.05.2024 um 17:59 +0200 schrieb Santiago Vila: > El 1/1/23 a las 16:55, Uecker, Martin escribió: > > I can apply the patch, but I do not have much time now. > > Is there some urgency? > > Hello. A lot of time passed without activity on this bug. > > In the meantime, I became member of debian-med, so in theory, > I could fix this myself via team upload. Would you prefer > that I take care of this myself that way? I wouldn't mind if you did. There are some other bugs which could easily be fixed with a new upload (i.e. with minor patches in the bug tracker) > > (I would also handle bart-cuda, also affected but not reported yet) bart-cuda needs to be updated to 0.9.00 which should be straightforward in principle but need more work and a binary upload. Otherwise, I would have more time in July to do some work. Martin > > (Yes, I still think it would be worthy to fix this in bullseye, > for posterity and before it becomes LTS). > > Thanks.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 1/1/23 a las 16:55, Uecker, Martin escribió: I can apply the patch, but I do not have much time now. Is there some urgency? Hello. A lot of time passed without activity on this bug. In the meantime, I became member of debian-med, so in theory, I could fix this myself via team upload. Would you prefer that I take care of this myself that way? (I would also handle bart-cuda, also affected but not reported yet) (Yes, I still think it would be worthy to fix this in bullseye, for posterity and before it becomes LTS). Thanks.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 1/1/23 a las 16:55, Uecker, Martin escribió: I can apply the patch, but I do not have much time now. Is there some urgency? Not really. There will be several more point releases of bullseye before it becomes LTS. I just hope that we can fix this before that. Thanks a lot.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Am 01.01.23 um 16:55 schrieb Uecker, Martin: This is likely a numerical error caused by reordering a floating point sum by parallelization. It is not worth spending time on it. I can apply the patch, but I do not have much time now. Is there some urgency? Not from my side, I just tried to help debugging the issue. Kind regards, Bernhard
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
> > > Am 01.01.23 um 15:55 schrieb Uecker, Martin: > > > > One could just relax (or simply remove) the test from bullseye > > or packport the version bookworm. > > I guess that would be applying 0003-relax-failing-unit-test.patch > to the Bullseye version. Yes, this should do it. > > > The wine code is broken (it violates the effective types > > rules of ISO C). > > Ok, just wanted to offer an option. > Yes, thanks. > Am 01.01.23 um 16:02 schrieb Santiago Vila: > > > Hi. Such failure rate differs a lot from what I get, which is > > about 50% in some systems (which is why I believe we should fix this > > in bullseye). > > > > Maybe this is an issue of tests optimized for Intel and failing > > a lot on AMD or viceversa (there was another package for which that > > happened). > > > If it might be of any help - my system is a "AMD Ryzen 7 1700", > the qemu VM runs with "-enable-kvm -cpu host -smp 16". > > By locking the process to just a single cpu I do not get any failures: > taskset -c 0 bash -c "while true; do ./test_nufft ; done" > > If I allow two cpus the failure rate is at 77%: > taskset -c 0,1 bash -c "while true; do ./test_nufft ; done" > > > This still does not affect a Bookworm VM, no failures even with > the relax patch removed. This is likely a numerical error caused by reordering a floating point sum by parallelization. It is not worth spending time on it. I can apply the patch, but I do not have much time now. Is there some urgency? Martin
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 1/1/23 a las 16:27, Bernhard Übelacker escribió: If it might be of any help - my system is a "AMD Ryzen 7 1700", the qemu VM runs with "-enable-kvm -cpu host -smp 16". By locking the process to just a single cpu I do not get any failures: taskset -c 0 bash -c "while true; do ./test_nufft ; done" If I allow two cpus the failure rate is at 77%: taskset -c 0,1 bash -c "while true; do ./test_nufft ; done" Aha, this matches what I get using Hetzner VMs: Works ok if the machine has a single CPU, and it fails a lot if it has 2 CPUs (I only test with 1 and 2 CPUs). So, maybe it has something to do with parallel processing. Thanks.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Hello, thanks for your immediate responses. Am 01.01.23 um 15:55 schrieb Uecker, Martin: One could just relax (or simply remove) the test from bullseye or packport the version bookworm. I guess that would be applying 0003-relax-failing-unit-test.patch to the Bullseye version. The wine code is broken (it violates the effective types rules of ISO C). Ok, just wanted to offer an option. Am 01.01.23 um 16:02 schrieb Santiago Vila: Hi. Such failure rate differs a lot from what I get, which is about 50% in some systems (which is why I believe we should fix this in bullseye). Maybe this is an issue of tests optimized for Intel and failing a lot on AMD or viceversa (there was another package for which that happened). If it might be of any help - my system is a "AMD Ryzen 7 1700", the qemu VM runs with "-enable-kvm -cpu host -smp 16". By locking the process to just a single cpu I do not get any failures: taskset -c 0 bash -c "while true; do ./test_nufft ; done" If I allow two cpus the failure rate is at 77%: taskset -c 0,1 bash -c "while true; do ./test_nufft ; done" This still does not affect a Bookworm VM, no failures even with the relax patch removed. Kind regards, Bernhard
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
El 1/1/23 a las 15:39, Bernhard Übelacker escribió: I could reproduce this failure in a bullseye VM. There the "test_nufft_adjoint" fails in about 1.2 % of the runs. Hi. Such failure rate differs a lot from what I get, which is about 50% in some systems (which is why I believe we should fix this in bullseye). Maybe this is an issue of tests optimized for Intel and failing a lot on AMD or viceversa (there was another package for which that happened). As I said in the bug report, I can offer a virtual machine where this happens 50% of the time (not 1.2%). Please contact me privately if you are interested. Thanks.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
One could just relax (or simply remove) the test from bullseye or packport the version bookworm. The wine code is broken (it violates the effective types rules of ISO C). Martin Am Sonntag, dem 01.01.2023 um 15:39 +0100 schrieb Bernhard Übelacker: > Dear Maintainer, > I could reproduce this failure in a bullseye VM. > There the "test_nufft_adjoint" fails in about 1.2 % of the runs. > > Attached diff helps to make it more visible. > > It looks like the float comparison fails because the > limit of "1.E-6f" is slightly not enough. > > If interpret following floating point comparison document right, > then the failing cases are just 8 representable floats "ULPs" > away from the expected value, below 8 it does not fail. > > > https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ > > Maybe upstream could consider changing that > float comparison to something like this: > > > https://source.winehq.org/git/wine.git/blob/HEAD:/dlls/ddraw/tests/ddraw7.c#l61 > > > In the newer Bookworm package I have found following patch, > which does relax exactly this test: > > > https://salsa.debian.org/med-team/bart/-/blob/master/debian/patches/0003-relax-failing-unit-test.patch > > But for some reason it does still not fail if I remove that > patch in the Bookworm version. > > Kind regards, > Bernhard > > > > make utest > while true; do ./test_nufft ; done > > > Bullseye/stable/bart-0.6.00: > - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198080e+01 > sc2=6.3619926397041880e+01 diff=9.6109602054639254e-07 diff_ulp=7 > adjoint diff: 0.01 9.6109602054639254e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed. > > - -1.067273+3.247030i - -1.067273+3.247030i - sc1=6.3619926397041830e+01 > sc2=6.3619926397041880e+01 diff=8.3446502685546875e-07 diff_ulp=7 > adjoint diff: 0.01 8.3446502685546875e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed. > > - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198087e+01 > sc2=6.3619926397041880e+01 diff=8.5963040419301251e-07 diff_ulp=6 > adjoint diff: 0.01 8.5963040419301251e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed. > > - -1.067272+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198073e+01 > sc2=6.3619926397041880e+01 diff=1.0662403155947686e-06 diff_ulp=8 > adjoint diff: 0.01 1.0662403155947686e-06, limit: 9.999747524271e-07 > ERROR: ./test_nufft: 0/ 1 passed. > > - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198087e+01 > sc2=6.3619926397041880e+01 diff=8.5963040419301251e-07 diff_ulp=6 > adjoint diff: 0.01 8.5963040419301251e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed. > > - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198080e+01 > sc2=6.3619926397041880e+01 diff=9.6109602054639254e-07 diff_ulp=7 > adjoint diff: 0.01 9.6109602054639254e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed. > > > Bookworm/testing/bart-0.8.00: > - -1.067272+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198073e+01 > sc2=6.3619926397041880e+01 diff=1.0662403155947686e-06 diff_ulp=8 > adjoint diff: 0.00 3.1195452265819767e-07, limit: 9.999747524271e-07 > ./test_nufft: 1/ 1 passed.
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Dear Maintainer, I could reproduce this failure in a bullseye VM. There the "test_nufft_adjoint" fails in about 1.2 % of the runs. Attached diff helps to make it more visible. It looks like the float comparison fails because the limit of "1.E-6f" is slightly not enough. If interpret following floating point comparison document right, then the failing cases are just 8 representable floats "ULPs" away from the expected value, below 8 it does not fail. https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/ Maybe upstream could consider changing that float comparison to something like this: https://source.winehq.org/git/wine.git/blob/HEAD:/dlls/ddraw/tests/ddraw7.c#l61 In the newer Bookworm package I have found following patch, which does relax exactly this test: https://salsa.debian.org/med-team/bart/-/blob/master/debian/patches/0003-relax-failing-unit-test.patch But for some reason it does still not fail if I remove that patch in the Bookworm version. Kind regards, Bernhard make utest while true; do ./test_nufft ; done Bullseye/stable/bart-0.6.00: - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198080e+01 sc2=6.3619926397041880e+01 diff=9.6109602054639254e-07 diff_ulp=7 adjoint diff: 0.01 9.6109602054639254e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed. - -1.067273+3.247030i - -1.067273+3.247030i - sc1=6.3619926397041830e+01 sc2=6.3619926397041880e+01 diff=8.3446502685546875e-07 diff_ulp=7 adjoint diff: 0.01 8.3446502685546875e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed. - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198087e+01 sc2=6.3619926397041880e+01 diff=8.5963040419301251e-07 diff_ulp=6 adjoint diff: 0.01 8.5963040419301251e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed. - -1.067272+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198073e+01 sc2=6.3619926397041880e+01 diff=1.0662403155947686e-06 diff_ulp=8 adjoint diff: 0.01 1.0662403155947686e-06, limit: 9.999747524271e-07 ERROR: ./test_nufft: 0/ 1 passed. - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198087e+01 sc2=6.3619926397041880e+01 diff=8.5963040419301251e-07 diff_ulp=6 adjoint diff: 0.01 8.5963040419301251e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed. - -1.067273+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198080e+01 sc2=6.3619926397041880e+01 diff=9.6109602054639254e-07 diff_ulp=7 adjoint diff: 0.01 9.6109602054639254e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed. Bookworm/testing/bart-0.8.00: - -1.067272+3.247031i - -1.067273+3.247030i - sc1=6.3619987432198073e+01 sc2=6.3619926397041880e+01 diff=1.0662403155947686e-06 diff_ulp=8 adjoint diff: 0.00 3.1195452265819767e-07, limit: 9.999747524271e-07 ./test_nufft: 1/ 1 passed.diff --git a/src/linops/lintest.c b/src/linops/lintest.c index c1d1ebc..b0e3bc1 100644 --- a/src/linops/lintest.c +++ b/src/linops/lintest.c @@ -59,7 +59,7 @@ static float linop_test_adjoint_generic(const struct linop_s* op, bool rvc) md_free(tmp3); md_free(tmp4); - debug_printf(DP_DEBUG4, "- %f%+fi - %f%+fi -\n", crealf(sc1), cimagf(sc1), crealf(sc2), cimagf(sc2)); + debug_printf(DP_INFO, "- %f%+fi - %f%+fi - sc1=%.16e sc2=%.16e diff=%.16e diff_ulp=%d\n", crealf(sc1), cimagf(sc1), crealf(sc2), cimagf(sc2), sc1, sc2, cabsf(sc1 - sc2), abs(*(int*) - *(int*))); return cabsf(sc1 - sc2); } diff --git a/utests/test_nufft.c b/utests/test_nufft.c index ec02762..54163dc 100644 --- a/utests/test_nufft.c +++ b/utests/test_nufft.c @@ -112,7 +112,7 @@ static bool test_nufft_adjoint(void) float diff = linop_test_adjoint(op); - debug_printf(DP_DEBUG1, "adjoint diff: %f\n", diff); + debug_printf(DP_INFO, "adjoint diff: %f %.16e, limit: %.16e\n", diff, diff, 1.E-6f); bool ret = (diff < 1.E-6f); @@ -231,13 +231,6 @@ static bool test_nufft_basis_toeplitz(void) -UT_REGISTER_TEST(test_nufft_forward); UT_REGISTER_TEST(test_nufft_adjoint); -UT_REGISTER_TEST(test_nufft_normal); -UT_REGISTER_TEST(test_nufft_toeplitz_weights); -UT_REGISTER_TEST(test_nufft_toeplitz_noweights); -UT_REGISTER_TEST(test_nufft_basis_adjoint); -UT_REGISTER_TEST(test_nufft_basis_normal); -UT_REGISTER_TEST(test_nufft_basis_toeplitz);
Bug#1026061: bart: FTBFS randomly in bullseye (failing test)
Package: src:bart Version: 0.6.00-3 Fixed: 0.8.00-3 Severity: serious Tags: ftbfs Dear maintainer: During a rebuild of all packages in bullseye, your package failed to build: [...] debian/rules binary-indep dh binary-indep dh_update_autotools_config -i dh_autoreconf -i dh_auto_configure -i dh_auto_build -i make -j2 "INSTALL=install --strip-program=true" make[1]: Entering directory '/<>' make MAKESTAGE=2 make[2]: Entering directory '/<>' make[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule. gcc -Wdate-time -D_FORTIFY_SOURCE=2 -MMD -MF /<>/src/.bart.d -iquote /<>/src/ -I/usr//include/ -I/usr//include -DFFTWTHREADS -DMAIN_LIST="avg, bench, bin, bitmask, cabs, caldir, calmat, carg, casorati, cc, ccapply, cdf97, circshift, conj, conv, copy, cpyphs, creal, crop, delta, ecalib, ecaltwo, estdelay, estdims, estshift, estvar, extract, fakeksp, fft, fftmod, fftrot, fftshift, filter, flatten, flip, fmac, homodyne, index, invert, itsense, join, looklocker, lrmatrix, mandelbrot, mip, moba, nlinv, noise, normalize, nrmse, nufft, ones, pattern, phantom, pics, pocsense, poisson, poly, repmat, reshape, resize, rmfreq, rof, rss, rtnlinv, sake, saxpy, scale, sdot, show, slice, spow, sqpics, squeeze, ssa, std, svd, tgv, threshold, toimg, traj, transpose, twixread, upat, var, vec, version, walsh, wave, wavelet, wavepsf, whiten, window, wshfl, zeros, zexp, ()" -include src/main.h -g -O2 -ffile-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -std=gnu11 -fopenmp -c -o /<>/src/bart.o /<>/src/bart.c [... snipped ...] gcc -Wdate-time -D_FORTIFY_SOURCE=2 -MMD -MF utests/.test_prox.d -iquote /<>/src/ -I/usr//include/ -I/usr//include -DFFTWTHREADS -g -O2 -ffile-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -std=gnu11 -fopenmp -c -o utests/test_prox.o utests/test_prox.c gcc -Wl,-z,relro -Wl,-z,now -rdynamic -rdynamic -g -O2 -ffile-prefix-map=/<>=. -fstack-protector-strong -Wformat -Werror=format-security -std=gnu11 -fopenmp -Wdate-time -D_FORTIFY_SOURCE=2 -MMD -MF ./.test_prox.d -iquote /<>/src/ -I/usr//include/ -I/usr//include -DFFTWTHREADS -DUTESTS="call_test_thresh, call_test_auto_norm," -o test_prox utests/utest.c utests/test_prox.o lib/libiter.a lib/liblinops.a lib/libnum.a lib/libmisc.a lib/libnum.a lib/libmisc.a -L/usr//lib -lfftw3f -lfftw3f_threads -L/usr//lib -llapacke -lblas -lm ./test_linop_matrix ./test_linop_matrix: 4/ 4 passed. ./test_linop ./test_linop: 3/ 3 passed. ./test_batchsvd ./test_batchsvd: 2/ 2 passed. ./test_pattern ./test_pattern: 1/ 1 passed. ./test_types ./test_types: 2/ 2 passed. ./test_misc ./test_misc: 2/ 2 passed. ./test_moba ./test_moba: 1/ 1 passed. ./test_nlop ./test_nlop: 15/15 passed. ./test_nufft [31mERROR: ./test_nufft: 7/ 8 passed. [0mmake[3]: *** [Makefile:685: utests-all] Error 1 make[3]: Leaving directory '/<>' make[2]: *** [Makefile:273: utest] Error 2 make[2]: Leaving directory '/<>' make[1]: *** [debian/rules:22: override_dh_auto_test] Error 2 make[1]: Leaving directory '/<>' make: *** [debian/rules:16: binary-indep] Error 2 dpkg-buildpackage: error: debian/rules binary-indep subprocess returned exit status 2 Note: The failure happens randomly, but the failure rate is around 70%, which is why I'm using a severity of "serious". In either case, I'm marking this as fixed in the bookworm version, as the failures do not seem to happen there. About the archive rebuild: The build was made using virtual machines from Hetzner, with enough memory, enough disk, and either one or two CPUs, using a reduced chroot with only build-essential packages (plus debhelper). If you could not reproduce the bug please contact me privately, as I am willing to provide ssh access to a virtual machine where the bug is fully reproducible. If this is really a bug in one of the build-depends, please use reassign and affects, so that this is still visible in the BTS web page for this package. Thanks.
