Re: Test suite fails when run in parallel for i686-linux
Hi Daniel, Daniel Sahlberg writes: > Den lör 8 apr. 2023 kl 21:18 skrev Maxim Cournoyer < > maxim.courno...@gmail.com>: > [...] > > Thanks for your help in verifying my environment! > > >> > I'm currently leaning towards something fishy with compiler/libraries >> used >> > in guix. I'm trying to figure out exactly how this goes wrong, but >> threads >> > programming is far from my comfort zone. In case anyone else would like >> to >> > pick up, feel free! >> >> I think the problem probably be reproducible from another i686-linux >> distribution; perhaps it could be tried using Docker for example. >> > > I've spent the afternoon setting up two virtual machines under Virtual Box > 6.1: > > - Debian 11.6 i386 (uname -a says "5.10.0-21-686 #1 SMP Debian 5.10.162-1 > (2023-01-21 i686 GNU/Linux") > > - Guix 1.4.0 i686 (uname -a says "6.0.10-gnu #1 SMP PREEMPTY_DYNAMIC 1 i686 > GNU/Linux") > > Both are assigned 6 CPUs each and 4GB memory. Both run APR 1.7.0 > > On the Debian machine, I've checked out Subversion trunk and on Guix I've > downloaded the distrubution tarball for 1.14.2. > > Both are configured with --enable-maintainer-mode and nothing else. On the > Debian machine I have gcc 10.2.1-6. On Guix, I did "guix install gcc@10.3". > I've also added libraries as needed to have configure complete > successfully. Unfortunately, I can't figure out some libraries/function > under Guix, for example Serf, Boost and APXS (for building mod_dav_svn). The easiest path to get a working environment to build Subversion with Guix would be: --8<---cut here---start->8--- guix shell --development subversion --8<---cut here---end--->8--- or to not share the host system libraries/commands: --8<---cut here---start->8--- guix shell --pure -D subversion --8<---cut here---end--->8--- or to run in a fully containerized shell process: --8<---cut here---start->8--- guix shell --container -D subversion --8<---cut here---end--->8--- Note that 'gcc' alone won't be sufficient to build, as you'll want the glibc to be there as well and the linker; there's a 'gcc-toolchain' package for that purpose. To view the package definition of the subversion package, you can invoke: --8<---cut here---start->8--- guix edit subversion --8<---cut here---end--->8--- And see if there is anything special done as a configure/pre-configure step to get it to build with Guix (but I don't see anything special myself). > On the good side: Under Guix, I can reproduce the same error as Maxim, > mostly the assert on put_xlate_handle_node. I've run the test suite 5 times > and only once was it successful. This at least proves there isn't a problem > in the way the Guix system download and configure Subversion. > > However under Debian, I can't see the same error. I've run the test suite > at least 5 times and not a single failure. > > I've checked apr.h on both systems and APR_HAS_THREADS is 1 on both of them. > > At this point, I can't figure out what else to do. To me it still seems > there is something strange with the Guix environment / distribution. > Obviously the tests above are not completely the same, for example because > of the differences in libraries I'm using (in particular, I can't see if > there was any specific options in the APR build). Thanks for going to lengths to verify this! It's curious that Debian i686 couldn't reproduce the problem... I'm out of ideas now as well. If you find something, I'll be interested to know! -- Thanks, Maxim
Re: Test suite fails when run in parallel for i686-linux
Den lör 8 apr. 2023 kl 21:18 skrev Maxim Cournoyer < maxim.courno...@gmail.com>: [...] Thanks for your help in verifying my environment! > > I'm currently leaning towards something fishy with compiler/libraries > used > > in guix. I'm trying to figure out exactly how this goes wrong, but > threads > > programming is far from my comfort zone. In case anyone else would like > to > > pick up, feel free! > > I think the problem probably be reproducible from another i686-linux > distribution; perhaps it could be tried using Docker for example. > I've spent the afternoon setting up two virtual machines under Virtual Box 6.1: - Debian 11.6 i386 (uname -a says "5.10.0-21-686 #1 SMP Debian 5.10.162-1 (2023-01-21 i686 GNU/Linux") - Guix 1.4.0 i686 (uname -a says "6.0.10-gnu #1 SMP PREEMPTY_DYNAMIC 1 i686 GNU/Linux") Both are assigned 6 CPUs each and 4GB memory. Both run APR 1.7.0 On the Debian machine, I've checked out Subversion trunk and on Guix I've downloaded the distrubution tarball for 1.14.2. Both are configured with --enable-maintainer-mode and nothing else. On the Debian machine I have gcc 10.2.1-6. On Guix, I did "guix install gcc@10.3". I've also added libraries as needed to have configure complete successfully. Unfortunately, I can't figure out some libraries/function under Guix, for example Serf, Boost and APXS (for building mod_dav_svn). On the good side: Under Guix, I can reproduce the same error as Maxim, mostly the assert on put_xlate_handle_node. I've run the test suite 5 times and only once was it successful. This at least proves there isn't a problem in the way the Guix system download and configure Subversion. However under Debian, I can't see the same error. I've run the test suite at least 5 times and not a single failure. I've checked apr.h on both systems and APR_HAS_THREADS is 1 on both of them. At this point, I can't figure out what else to do. To me it still seems there is something strange with the Guix environment / distribution. Obviously the tests above are not completely the same, for example because of the differences in libraries I'm using (in particular, I can't see if there was any specific options in the APR build). Kind regards, Daniel
Re: Test suite fails when run in parallel for i686-linux
Hi Daniel, Daniel Sahlberg writes: > Den mån 3 apr. 2023 kl 17:35 skrev Maxim Cournoyer < > maxim.courno...@gmail.com>: > >> Oh. This suggests your installation does not make use of the available >> binary substitutes that would dramatically speed things up and >> workaround the bug you encountered (which is caused by TLS certificates >> expiring in the OpenSSL test suite, see: >> https://issues.guix.gnu.org/56137). >> > > I used the --no-substitutes argument as suggested. Maybe this is the reason? Ah yes, silly me! It would have been better advice to simply suggest to try to build it without that --no-substitutes argument, apologies! > The substitute servers key should have been authorized at installation >> by the guix-install.sh script, and Guix as of 1.4.0 uses both >> 'https://ci.guix.gnu.org' and 'https://bordeaux.guix.gnu.org' as its >> default substitute providers. >> >> What does 'cat /etc/guix/acl' show? It should have something like, the >> first entry being Bordeaux and the second one for Berlin. >> >> --8<---cut here---start->8--- >> (acl >> (entry > > > [...] > > ) >> ) >> --8<---cut here---end--->8--- >> > > I have the same keys, only in opposite order. Your setup appears to be fine :-). [...] > In the meantime, I manually set the date/time to 2022-01-01 and that seemed > to do the trick. The build completed successfully. > > In the Subversion build/test I see the same errors you have. I then did: > > [[[ > $ guix build --no-substitutes --no-grafts --system=i686-linux --keep-failed > subversion > ... > $ cd /tmp/guix-build-subversion-1.14.2.drv-0/ > $ source environment-variables > $ cd subversion-1.14.2 > $ make check -j4 PARALLEL=4 > ]]] > > This is now a functional environment to do more tests. I've been able to > reproduce the error in this environment. > > If I did the same, without sourcing the environment variables (I had to > make clean; ./config.nice before make), ie using the compiler and libraries > included with the distribution, the tests succeed. Right; you wouldn't be using the i686 libraries/toolchain then though (it'd build for x86_64, which is not known to have the problem). > I'm currently leaning towards something fishy with compiler/libraries used > in guix. I'm trying to figure out exactly how this goes wrong, but threads > programming is far from my comfort zone. In case anyone else would like to > pick up, feel free! I think the problem probably be reproducible from another i686-linux distribution; perhaps it could be tried using Docker for example. -- Thanks, Maxim
Re: Test suite fails when run in parallel for i686-linux
Den mån 3 apr. 2023 kl 17:35 skrev Maxim Cournoyer < maxim.courno...@gmail.com>: > Oh. This suggests your installation does not make use of the available > binary substitutes that would dramatically speed things up and > workaround the bug you encountered (which is caused by TLS certificates > expiring in the OpenSSL test suite, see: > https://issues.guix.gnu.org/56137). > I used the --no-substitutes argument as suggested. Maybe this is the reason? The substitute servers key should have been authorized at installation > by the guix-install.sh script, and Guix as of 1.4.0 uses both > 'https://ci.guix.gnu.org' and 'https://bordeaux.guix.gnu.org' as its > default substitute providers. > > What does 'cat /etc/guix/acl' show? It should have something like, the > first entry being Bordeaux and the second one for Berlin. > > --8<---cut here---start->8--- > (acl > (entry [...] ) > ) > --8<---cut here---end--->8--- > I have the same keys, only in opposite order. > 'guix weather openssl' should show it being available. I'd also like to > see the output of 'guix describe', to know from on which commit your > Guix is at. > dsg@dsg-ubuntu2210:~$ guix weather openssl hint: Consider installing the `glibc-locales' package and defining `GUIX_LOCPATH', along these lines: guix install glibc-locales export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale" See the "Application Setup" section in the manual, for more info. computing 1 package derivations for x86_64-linux... looking for 3 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org ☀ 100.0% substitutes available (3 out of 3) at least 11.4 MiB of nars (compressed) 32.6 MiB on disk (uncompressed) 0.977 seconds per request (2.9 seconds in total) 1.0 requests per second at least 1,000 queued builds aarch64-linux: 947 (94.7%) powerpc64le-linux: 51 (5.1%) i686-linux: 2 (.2%) build rate: 13.32 builds per hour i686-linux: 5.36 builds per hour x86_64-linux: 4.97 builds per hour aarch64-linux: 2.18 builds per hour powerpc64le-linux: 0.83 builds per hour armhf-linux: 0.04 builds per hour looking for 3 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org ☀ 100.0% substitutes available (3 out of 3) at least 7.8 MiB of nars (compressed) 32.6 MiB on disk (uncompressed) 0.244 seconds per request (0.7 seconds in total) 4.1 requests per second (continuous integration information unavailable) dsg@dsg-ubuntu2210:~$ guix describe hint: Consider installing the `glibc-locales' package and defining `GUIX_LOCPATH', along these lines: guix install glibc-locales export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale" See the "Application Setup" section in the manual, for more info. Generation 1Apr 02 2023 09:27:21(current) guix f0c8cb4 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: f0c8cb44a4d912263025616bc921c011fcb1f0b4 If it isn't try with --substitute-urls; perhaps the default ones differ > for some reason: > --8<---cut here---start->8--- > guix weather openssl \ > --substitute-urls='https://bordeaux.guix.gnu.org https://ci.guix.gnu.org > ' > --8<---cut here---end--->8--- > In the meantime, I manually set the date/time to 2022-01-01 and that seemed to do the trick. The build completed successfully. In the Subversion build/test I see the same errors you have. I then did: [[[ $ guix build --no-substitutes --no-grafts --system=i686-linux --keep-failed subversion ... $ cd /tmp/guix-build-subversion-1.14.2.drv-0/ $ source environment-variables $ cd subversion-1.14.2 $ make check -j4 PARALLEL=4 ]]] This is now a functional environment to do more tests. I've been able to reproduce the error in this environment. If I did the same, without sourcing the environment variables (I had to make clean; ./config.nice before make), ie using the compiler and libraries included with the distribution, the tests succeed. I'm currently leaning towards something fishy with compiler/libraries used in guix. I'm trying to figure out exactly how this goes wrong, but threads programming is far from my comfort zone. In case anyone else would like to pick up, feel free! Kind regards, Daniel
Re: Test suite fails when run in parallel for i686-linux
Hi, Daniel Sahlberg writes: > Den sön 2 apr. 2023 kl 05:01 skrev Maxim Cournoyer < > maxim.courno...@gmail.com>: > [...] > >> The OS is Guix System; I'm currently running the 5.15.103 kernel. I was >> testing on the staging branch, which has 1.14.2. Assuming you have Guix >> installed on your GNU/Linux system: >> >> --8<---cut here---start->8--- >> wget -O guix-install.sh https://guix-install.sh >> chmod +x guix-insall.sh >> yes | sudo ./guix-install.sh >> # Quit, then reopen your session >> guix pull # update Guix itself >> --8<---cut here---end--->8--- >> >> You should be able to reproduce on a x86_64 system with: >> >> --8<---cut here---start->8--- >> guix build --no-substitutes --no-grafts --system=i686-linux subversion >> --8<---cut here---end--->8--- >> > > Now this is slightly off-topic, but since I'l like to see the reproduction > of this error, I spent the better part of the day waiting for the build to > complete, however run into problems with openssl test 80-test_ssl_new.t. > Any ideas how to get past that one? Oh. This suggests your installation does not make use of the available binary substitutes that would dramatically speed things up and workaround the bug you encountered (which is caused by TLS certificates expiring in the OpenSSL test suite, see: https://issues.guix.gnu.org/56137). The substitute servers key should have been authorized at installation by the guix-install.sh script, and Guix as of 1.4.0 uses both 'https://ci.guix.gnu.org' and 'https://bordeaux.guix.gnu.org' as its default substitute providers. What does 'cat /etc/guix/acl' show? It should have something like, the first entry being Bordeaux and the second one for Berlin. --8<---cut here---start->8--- (acl (entry (public-key (ecc (curve Ed25519) (q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#) ) ) (tag (guix import) ) ) (entry (public-key (ecc (curve Ed25519) (q #7D602902D3A2DBB83F8A0FB98602A754C5493B0B778C8D1DD4E0F41DE14DE34F#) ) ) (tag (guix import) ) ) ) --8<---cut here---end--->8--- 'guix weather openssl' should show it being available. I'd also like to see the output of 'guix describe', to know from on which commit your Guix is at. If it isn't try with --substitute-urls; perhaps the default ones differ for some reason: --8<---cut here---start->8--- guix weather openssl \ --substitute-urls='https://bordeaux.guix.gnu.org https://ci.guix.gnu.org' --8<---cut here---end--->8--- Thanks for your efforts! -- Thanks, Maxim
Re: Test suite fails when run in parallel for i686-linux
Den sön 2 apr. 2023 kl 05:01 skrev Maxim Cournoyer < maxim.courno...@gmail.com>: [...] > The OS is Guix System; I'm currently running the 5.15.103 kernel. I was > testing on the staging branch, which has 1.14.2. Assuming you have Guix > installed on your GNU/Linux system: > > --8<---cut here---start->8--- > wget -O guix-install.sh https://guix-install.sh > chmod +x guix-insall.sh > yes | sudo ./guix-install.sh > # Quit, then reopen your session > guix pull # update Guix itself > --8<---cut here---end--->8--- > > You should be able to reproduce on a x86_64 system with: > > --8<---cut here---start->8--- > guix build --no-substitutes --no-grafts --system=i686-linux subversion > --8<---cut here---end--->8--- > Now this is slightly off-topic, but since I'l like to see the reproduction of this error, I spent the better part of the day waiting for the build to complete, however run into problems with openssl test 80-test_ssl_new.t. Any ideas how to get past that one? Kind regards, Daniel Sahlberg
Re: Test suite fails when run in parallel for i686-linux
Hi! Daniel Sahlberg writes: > Thanks for the tests.log! I've tried to look into it and sending some > comments below! > > Den fre 31 mars 2023 kl 23:35 skrev Maxim Cournoyer < > maxim.courno...@gmail.com>: > >> Hello, >> >> Nathan Hartman writes: >> >> > On Thu, Mar 30, 2023 at 11:13 PM Maxim Cournoyer >> > wrote: >> > > [...] > > I assume the log is from this build: > > >> And on an old 4 cores desktop: Yes, I confirm it's from this build. >> --8<---cut here---start->8--- >> At least one test FAILED, checking >> /tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/tests.log >> FAIL: fs-fs-pack-test: Unknown test failure (-6); see tests.log. >> > > fs-fs-pack-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: > Assertion `node->next == NULL' failed. > > >> FAIL: fs-test: Unknown test failure (-6); see tests.log. >> > > fs-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion > `node->next == NULL' failed. > > >> FAIL: io-test: Unknown test failure (-6); see tests.log. >> > > io-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion > `node->next == NULL' failed. > > >> FAIL: op-depth-test: Unknown test failure (-6); see tests.log. >> > > op-depth-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: > Assertion `node->next == NULL' failed. > > There is some mutex handling in utf.c. I'm thinking if APR is missing the > appropriate mutex handling in some weird way. What is the exact versions of > APR/APR-util in your builld? > > FAIL: repos-test 2: test deletions under copies in node_tree code >> > > svn_tests: E02: Repository creation failed > svn_tests: E02: Creating db lock file > svn_tests: E02: Can'FAIL: repos-test 1: test svn_repos_dir_delta2 > svn_tests: E02: Repository creation failed > svn_tests: E02: Creating db logs lock file > svn_tests: E02: Can't open file > '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-del-under-copy/locks/db-logs.lock': > No such file or directory > FAIL: repos-test 2: test deletions under copies in node_tree code > > >> FAIL: repos-test 3: test svn_repos_history() (partially) >> > > svn_tests: E02: Repository creation failed > svn_tests: E02: Creating lock dir > svn_tests: E02: Can't create directory > '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-revisions-changed/locks': > No such file or directory > FAIL: repos-test 3: test svn_repos_history() (partially) > > FAIL: repos-test 5: test svn_repos_node_locations some more >> > > svn_tests: E02: Repository creation failed > svn_tests: E02: Creating pre-commit hook > svn_tests: E02: Can't change perms of file > '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-node-locations2/hooks/pre-commit.tmpl': > No such file or directory > FAIL: repos-test 5: test svn_repos_node_locations some more > > >> FAIL: repos-test: Unknown test failure (-11); see tests.log. >> > > Since each test operate on its own WC, I can't really figure out why there > would be issues with these tests, unless there is an error in > svn_atomic_inc() (see subversion/tests/svn_test_main.c, test_thread()) > causing the same test to be started more than once (by separate threads). > > >> So the failures appear to be non-deterministic, although some test suite >> such as repos-test fail more often than not. >> > > What is the exact OS (distribution/release/kernel version) you are running? The OS is Guix System; I'm currently running the 5.15.103 kernel. I was testing on the staging branch, which has 1.14.2. Assuming you have Guix installed on your GNU/Linux system: --8<---cut here---start->8--- wget -O guix-install.sh https://guix-install.sh chmod +x guix-insall.sh yes | sudo ./guix-install.sh # Quit, then reopen your session guix pull # update Guix itself --8<---cut here---end--->8--- You should be able to reproduce on a x86_64 system with: --8<---cut here---start->8--- guix build --no-substitutes --no-grafts --system=i686-linux subversion --8<---cut here---end--->8--- -- Thanks, Maxim
Re: Test suite fails when run in parallel for i686-linux
Thanks for the tests.log! I've tried to look into it and sending some comments below! Den fre 31 mars 2023 kl 23:35 skrev Maxim Cournoyer < maxim.courno...@gmail.com>: > Hello, > > Nathan Hartman writes: > > > On Thu, Mar 30, 2023 at 11:13 PM Maxim Cournoyer > > wrote: > [...] I assume the log is from this build: > And on an old 4 cores desktop: > > --8<---cut here---start->8--- > At least one test FAILED, checking > /tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/tests.log > FAIL: fs-fs-pack-test: Unknown test failure (-6); see tests.log. > fs-fs-pack-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion `node->next == NULL' failed. > FAIL: fs-test: Unknown test failure (-6); see tests.log. > fs-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion `node->next == NULL' failed. > FAIL: io-test: Unknown test failure (-6); see tests.log. > io-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion `node->next == NULL' failed. > FAIL: op-depth-test: Unknown test failure (-6); see tests.log. > op-depth-test: subversion/libsvn_subr/utf.c:424: put_xlate_handle_node: Assertion `node->next == NULL' failed. There is some mutex handling in utf.c. I'm thinking if APR is missing the appropriate mutex handling in some weird way. What is the exact versions of APR/APR-util in your builld? FAIL: repos-test 2: test deletions under copies in node_tree code > svn_tests: E02: Repository creation failed svn_tests: E02: Creating db lock file svn_tests: E02: Can'FAIL: repos-test 1: test svn_repos_dir_delta2 svn_tests: E02: Repository creation failed svn_tests: E02: Creating db logs lock file svn_tests: E02: Can't open file '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-del-under-copy/locks/db-logs.lock': No such file or directory FAIL: repos-test 2: test deletions under copies in node_tree code > FAIL: repos-test 3: test svn_repos_history() (partially) > svn_tests: E02: Repository creation failed svn_tests: E02: Creating lock dir svn_tests: E02: Can't create directory '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-revisions-changed/locks': No such file or directory FAIL: repos-test 3: test svn_repos_history() (partially) FAIL: repos-test 5: test svn_repos_node_locations some more > svn_tests: E02: Repository creation failed svn_tests: E02: Creating pre-commit hook svn_tests: E02: Can't change perms of file '/tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/subversion/tests/libsvn_repos/test-repo-node-locations2/hooks/pre-commit.tmpl': No such file or directory FAIL: repos-test 5: test svn_repos_node_locations some more > FAIL: repos-test: Unknown test failure (-11); see tests.log. > Since each test operate on its own WC, I can't really figure out why there would be issues with these tests, unless there is an error in svn_atomic_inc() (see subversion/tests/svn_test_main.c, test_thread()) causing the same test to be started more than once (by separate threads). > So the failures appear to be non-deterministic, although some test suite > such as repos-test fail more often than not. > What is the exact OS (distribution/release/kernel version) you are running? Kind regards, Daniel Sahlberg
Re: Test suite fails when run in parallel for i686-linux
On Thu, Mar 30, 2023 at 11:13 PM Maxim Cournoyer
wrote:
>
> Hello,
>
> This has been the case for the last 3 years, so I thought it'd be nice
> to report it, toward having it fixed! It seems to affect only the
> i686-linux machine, and not x86_64-linux.
Interesting. So, it fails on 32-bit, passes on 64-bit.
Continuing inline below...
> The test suite in run in parallel via the -j option of Make, e.g.
>
> 'make -j24 check'
>
> The PARALLEL environment variable is also set to the same value (e.g.:
> 24) before the test suite runs.
I noticed this from near the end of your output:
> Test suite failed, dumping logs.
> error: in phase 'check': uncaught exception:
> %exception #< program: "make" arguments: ("check" "-j" "16")
> exit-status: 2 term-signal: #f stop-signal: #f>
> phase `check' failed after 193.9 seconds
> command "make" "check" "-j" "16" failed with status 2
Notice the "check" "-j" "16" -- but you requested -j24. Is it possible
that a mismatch here is contributing to the failures? Could you try
'make -j16 check' and see if the results are any different?
>From earlier in your output:
> At least one test FAILED, checking
> /tmp/guix-build-subversion-1.14.2.drv-0/subversion-1.14.2/tests.log
> FAIL: client-test: Unknown test failure (-6); see tests.log.
> FAIL: diff-diff3-test: Unknown test failure (-6); see tests.log.
> FAIL: fs-fs-pack-test: Unknown test failure (-6); see tests.log.
> FAIL: fs-fs-private-test: Unknown test failure (-6); see tests.log.
> FAIL: fs-test: Unknown test failure (-6); see tests.log.
> FAIL: io-test: Unknown test failure (-6); see tests.log.
> FAIL: mtcc-test: Unknown test failure (-6); see tests.log.
> FAIL: op-depth-test: Unknown test failure (-6); see tests.log.
> FAIL: ra-test: Unknown test failure (-6); see tests.log.
> FAIL: repos-test: Unknown test failure (-6); see tests.log.
> Summary of test results:
> 2237 tests PASSED
> 165 tests SKIPPED
> 65 tests XFAILED (16 WORK-IN-PROGRESS)
> 10 tests FAILED
> Python version: 3.9.9.
> SUMMARY: Some tests failed
Do you happen to have the tests.log file mentioned above? It might
contain more details about the failures (or it might just tell us
about the same uncaught exception).
Finally, do the same tests fail on each parallel run or are there
variations?
Thanks for the report,
Nathan
