bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Mark H Weaver  writes:

> Hi Marius,
>
> Marius wrote:
>> There has been some activity around TokuDB in later versions of MariaDB,
>> maybe we can enable it again with 10.4.  For now, I think we should just
>> disable it.
>
> Disabling TokuDB for now sounds like a fine option.
> Thanks very much for looking into it.

Done in bba7a77ed9ad826bcdc6d9b8a183d66a23229501.

Thanks for reporting the issue.

Thinking forward, to trim the mariadb package, maybe it's possible to
build all plugins as separate derivations, and let the user choose a
union when setting up the service.


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Hi Marius,

Marius wrote:
> There has been some activity around TokuDB in later versions of MariaDB,
> maybe we can enable it again with 10.4.  For now, I think we should just
> disable it.

Disabling TokuDB for now sounds like a fine option.
Thanks very much for looking into it.

  Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Mark H Weaver  writes:

> Hello again,
>
>> Could it be that you don't have enough disk space for this test?  Do you
>> have the log file available still?
>
> I made another build attempt on my X200, this time logging the output of
> "df --si" every 10 seconds.  The free space started at ~11 GB free and
> never went below 7 GB, but the 'tokudb_bugs.mdev4533' test failed as
> before: "No space left on device" while trying to create the 'test'
> table.

Thanks for testing.  Out of curiousity I tried to enable TokuDB on my
server:

MariaDB [(none)]> INSTALL PLUGIN tokudb SONAME 'ha_tokudb';
ERROR 2006 (HY000): MySQL server has gone away

Ouch.

Unfortunately the Guix service does not seem to enable any kind of
logging, so I haven't dug further.  Loading other plugins seems to work
though.

I am currently trying this patch on Berlin:

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 6bfeaad9a2..c17031bb2c 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -659,6 +659,10 @@ Language.")
  ;; For now, disable the features that that use libarchive (xtrabackup).
  "-DWITH_LIBARCHIVE=OFF"
 
+ ;; FIXME: Disable the TokuDB engine, because its test suite frequently
+ ;; fails, and loading it crashes the server: .
+ "-DTOKUDB_OK=OFF"
+
  ;; Ensure the system libraries are used.
  "-DWITH_JEMALLOC=yes"
  "-DWITH_PCRE=system"
@@ -706,9 +710,6 @@ Language.")
   ;; 2030-12-31.  See  for details.
   "main.mysqldump"
 
-  ;; XXX: Fails sporadically.
-  "innodb_fts.crash_recovery"
-
   ;; FIXME: This test fails on i686:
   ;; -myisampack: Can't create/write to file (Errcode: 17 "File exists")
   ;; +myisampack: Can't create/write to file (Errcode: 17 "File exists)
@@ -786,7 +787,6 @@ Language.")
("libxml2" ,libxml2)
("ncurses" ,ncurses)
("pcre" ,pcre)
-   ("snappy" ,snappy)
("xz" ,xz)
("zlib" ,zlib)))
 (propagated-inputs

WDYT?

There has been some activity around TokuDB in later versions of MariaDB,
maybe we can enable it again with 10.4.  For now, I think we should just
disable it.


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Gábor Boskovits]

Hello,

Arne Babenhauserheide  ezt írta (időpont: 2019. júl. 14.,
Vas 13:11):

> Hi Mark,
>
> Mark H Weaver  writes:
> > My log file contains the same error in the 'tokudb_bugs.mdev4533' test:
> >
> >   mysqltest: At line 6: query 'CREATE TABLE t1 (a INT(11), b CHAR(8))
> ENGINE=TokuDB' failed: 1005: Can't create table `test`.`t1` (errno: 28 "No
> space left on device")
>

Could you test using df -i if the file system is not running out of inodes?
That is another reason when the no space left on device error is reported.

>
> > After the build attempt, the failed build directory is ~3.4 GB, and I
> > still have ~7.4 GB.  That seems to imply that I had over 10 GB free
> > before starting the build, which sounds about right.  I don't have a
> > separate /tmp partition.
> …
> > I should mention that I'm using Btrfs.
>
> I use ext4, but I saw no space left on device errors when running guix
> lint. Since I had 700GiB free, that does not sound like real missing
> disk space, but rather that something else is wrong.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>
Best regards,
g_bor

>

bug#35521: Mariadb test suite failures on x86_64-linux

[Arne Babenhauserheide]

Hi Mark,

Mark H Weaver  writes:
> My log file contains the same error in the 'tokudb_bugs.mdev4533' test:
>
>   mysqltest: At line 6: query 'CREATE TABLE t1 (a INT(11), b CHAR(8)) 
> ENGINE=TokuDB' failed: 1005: Can't create table `test`.`t1` (errno: 28 "No 
> space left on device")
>
> After the build attempt, the failed build directory is ~3.4 GB, and I
> still have ~7.4 GB.  That seems to imply that I had over 10 GB free
> before starting the build, which sounds about right.  I don't have a
> separate /tmp partition.
…
> I should mention that I'm using Btrfs.

I use ext4, but I saw no space left on device errors when running guix
lint. Since I had 700GiB free, that does not sound like real missing
disk space, but rather that something else is wrong.

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Hello again,

> Could it be that you don't have enough disk space for this test?  Do you
> have the log file available still?

I made another build attempt on my X200, this time logging the output of
"df --si" every 10 seconds.  The free space started at ~11 GB free and
never went below 7 GB, but the 'tokudb_bugs.mdev4533' test failed as
before: "No space left on device" while trying to create the 'test'
table.

  Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Hi Marius,

> Could it be that you don't have enough disk space for this test?  Do you
> have the log file available still?

Yes, I have not only the log file, but also the failed build directory.
My log file contains the same error in the 'tokudb_bugs.mdev4533' test:

  mysqltest: At line 6: query 'CREATE TABLE t1 (a INT(11), b CHAR(8)) 
ENGINE=TokuDB' failed: 1005: Can't create table `test`.`t1` (errno: 28 "No 
space left on device")

After the build attempt, the failed build directory is ~3.4 GB, and I
still have ~7.4 GB.  That seems to imply that I had over 10 GB free
before starting the build, which sounds about right.  I don't have a
separate /tmp partition.

I will make another build attempt, and this time I will watch the disk
utilization over time while the test suite is in progress.

I should mention that I'm using Btrfs.

  Thanks,
Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Mark H Weaver  writes:

> Hi Marius,
>
>> Update: Berlin built mariadb twice on core-updates with this patch:
>>
>> --8<---cut here---start->8---
>> diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
>> index 6bfeaad9a2..64bc0938b6 100644
>> --- a/gnu/packages/databases.scm
>> +++ b/gnu/packages/databases.scm
>> @@ -753,7 +753,7 @@ Language.")
>>   (with-directory-excursion "mysql-test"
>> (invoke "./mtr" "--verbose"
>> "--retry=3"
>> -   "--testcase-timeout=40"
>> +   "--testcase-timeout=80"
>> "--suite-timeout=600"
>> "--parallel" (number->string 
>> (parallel-job-count))
>> "--skip-test-list=unstable-tests"))
>> --8<---cut here---end--->8---
>>
>> Mark, Chris: Can you try this change with MariaDB 10.1.40 and see if it
>> works for you?
>
> I tried it, but it made no difference on my Thinkpad X200, which still
> fails the same way as before with 10.1.38:
>
>   Failing test(s): tokudb_bugs.mdev4533

I was about to push this patch to core-updates:

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 6bfeaad9a2..5d256b1af2 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -706,9 +706,6 @@ Language.")
   ;; 2030-12-31.  See  for details.
   "main.mysqldump"
 
-  ;; XXX: Fails sporadically.
-  "innodb_fts.crash_recovery"
-
   ;; FIXME: This test fails on i686:
   ;; -myisampack: Can't create/write to file (Errcode: 17 "File exists")
   ;; +myisampack: Can't create/write to file (Errcode: 17 "File exists)
@@ -753,7 +750,10 @@ Language.")
  (with-directory-excursion "mysql-test"
(invoke "./mtr" "--verbose"
"--retry=3"
-   "--testcase-timeout=40"
+   ;; On x86_64 we need a long timeout because of the
+   ;; TokuDB engine, whose individual test cases often
+   ;; require more than 1 hour to complete on busy hosts.
+   "--testcase-timeout=90"
"--suite-timeout=600"
"--parallel" (number->string (parallel-job-count))
"--skip-test-list=unstable-tests"))

Lo and behold, tokudb_bugs.mdev4533 failed when I tried it on Berlin.

A couple of lines above "Failing test(s):" is the test output:

--8<---cut here---start->8---
CURRENT_TEST: tokudb_bugs.mdev4533  
   
safe_process[29262]: parent_pid: 23338  
   
safe_process[29262]: Started child 29263, terminated: 0 
   
mysqltest: At line 6: query 'CREATE TABLE t1 (a INT(11), b CHAR(8)) 
ENGINE=TokuDB' failed: 1005: Ca
n't create table `test`.`t1` (errno: 28 "No space left on device")

The result from queries just before the failure was:
DROP TABLE IF EXISTS t1; 
CREATE TABLE t1 (a INT(11), b CHAR(8)) ENGINE=TokuDB;
safe_process[29262]: Got signal 17, child_pid: 29263
safe_process[29262]: Killing child: 29263
safe_process[29262]: Child exit: 1
--8<---cut here---end--->8---

Could it be that you don't have enough disk space for this test?  Do you
have the log file available still?

Here is the test in question:
https://github.com/MariaDB/server/blob/10.1/storage/tokudb/mysql-test/tokudb_bugs/t/mdev4533.test

As a side note, MariaDB is ~30 MiB bigger on x86_64 because of TokuDB.
It would be great to move it to a separate output.


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Earlier, I wrote:

>> Mark, Chris: Can you try this change with MariaDB 10.1.40 and see if it
>> works for you?
>
> I tried it, but it made no difference on my Thinkpad X200, which still
> fails the same way as before with 10.1.38:
>
>   Failing test(s): tokudb_bugs.mdev4533

I should clarify that I tested 10.1.40 this time, and it failed in the
same way that 10.1.38 failed for me before.

   Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Hi Marius,

> Update: Berlin built mariadb twice on core-updates with this patch:
>
> --8<---cut here---start->8---
> diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
> index 6bfeaad9a2..64bc0938b6 100644
> --- a/gnu/packages/databases.scm
> +++ b/gnu/packages/databases.scm
> @@ -753,7 +753,7 @@ Language.")
>   (with-directory-excursion "mysql-test"
> (invoke "./mtr" "--verbose"
> "--retry=3"
> -   "--testcase-timeout=40"
> +   "--testcase-timeout=80"
> "--suite-timeout=600"
> "--parallel" (number->string (parallel-job-count))
> "--skip-test-list=unstable-tests"))
> --8<---cut here---end--->8---
>
> Mark, Chris: Can you try this change with MariaDB 10.1.40 and see if it
> works for you?

I tried it, but it made no difference on my Thinkpad X200, which still
fails the same way as before with 10.1.38:

  Failing test(s): tokudb_bugs.mdev4533

Anyway, based on Giovanni's observations,

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35521#32

I'm now inclined to agree that these are likely to be flaky tests, so I
withdraw my objections to disabling them, in this specific case.

Having said that, I disagree with Giovanni's dismissal of my concerns in
general, here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35521#29

I will respond to that dismissal in a later message.

 Thanks,
   Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Marius Bakke  writes:

> Mark H Weaver  writes:
>
>> Hi,
>>
>> Marius Bakke  writes:
>>
>>> Chris Marusich  writes:
>>>
 Hi,

 I've been encountering this failure off and on for a few weeks now, and
 I'd like to help fix it.  In short, it seems like non-deterministic test
 failures, to me.  I think we should gather data and report the issue
 upstream, and maybe disable the offending tests in the meantime.
>>>
>>> I agree.  I notice many of these failing tests are for the TokuDB
>>> backend, which I doubt anyone is using in Guix anyway.
>>>
>>> Here is a patch that disables all tests mentioned in this report.  I
>>> would like to push it to core-updates.  Are there others?
>>
>> I'm concerned by how frequently and casually we simply disable failing
>> tests.  What is the utility of running test suites at all, if this is
>> how we respond?
>
> I had no idea this issue was so widespread until I noticed Berlins
> builders hit it more often than not.  I have not been able to reproduce
> these failures on my machines.  So it was kind of a panic reaction,
> being the person responsible for running these tests and all.
>
> Looking further into the changes between 10.1.37 and 10.1.38, I notice
> the 'tokudb.*' tests were enabled:
>
> https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51
>
> Watching the build on Berlin in real time, I also see that the test
> output grind nearly to a halt while running those.
> 'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if
> I'm reading the test output correctly.
>
> The default test case timeout is 40 minutes (as specified in the Guix
> package), but I'm using 80 for this build (60 was insufficient).
>
> I suspect the problem is that the 'tokudb.*' tests put a lot of strain
> on the file system, which causes these other tests to fail.  It's
> interesting that disabling parallel build was insufficient though.

Update: Berlin built mariadb twice on core-updates with this patch:

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 6bfeaad9a2..64bc0938b6 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -753,7 +753,7 @@ Language.")
  (with-directory-excursion "mysql-test"
(invoke "./mtr" "--verbose"
"--retry=3"
-   "--testcase-timeout=40"
+   "--testcase-timeout=80"
"--suite-timeout=600"
"--parallel" (number->string (parallel-job-count))
"--skip-test-list=unstable-tests"))

Mark, Chris: Can you try this change with MariaDB 10.1.40 and see if it
works for you?


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Giovanni Biscuolo]

Hi Marius,

Marius Bakke  writes:

[...]

> Looking further into the changes between 10.1.37 and 10.1.38, I notice
> the 'tokudb.*' tests were enabled:
>
> https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51

The very first thing I noticed lookng at that commit is it's subject:
"Updated list of unstable tests for 10.1.38 release"

The first comments of that file states:

--8<---cut here---start->8---
#  List the test cases which, unlike tests from disabled.def files,
#  can still be run on the current tree meaningfully, but are known
#  or suspected to fail sporadically on different reasons.
#
#  Most common reasons are either test failures observed in buildbot,
#  or recent modifications to the tests which make their stability
#  unknown.
#
#  Tests included due to recent modifications are later removed from the
#  list, if during a certain period they do not fail (and are not
#  modified again). Tests included due to intermittent failures are
#  removed when corresponding bug reports are closed.
#
#  Separate the test case name and the comment with ':'.
#
#. : MDEV-x - 
#
#  '*' wildcard in testcase names is supported.
#
#  To use the list, run MTR with --skip-test-list=unstable-tests option.
--8<---cut here---end--->8---

So *all* those rests _are_ considered unstable upstream.

IMHO they should be *selectively* skipped when they causes build
problems in Guix, including non-deterministic frequent ones like in this
case.

> Watching the build on Berlin in real time, I also see that the test
> output grind nearly to a halt while running those.
> 'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if
> I'm reading the test output correctly.

The same is happening upstream:
https://jira.mariadb.org/browse/MDEV-15198
https://jira.mariadb.org/browse/MDEV-16040 (duplicate of the above)
https://jira.mariadb.org/browse/MDEV-15271

That bugs (and all others related to unstable tests) are currently
unresolved.

HTH! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Giovanni Biscuolo]

Hi all,

for what it counts I 

Ludovic Courtès  writes:

> Mark H Weaver  skribis:

[...]

>> I'm concerned by how frequently and casually we simply disable failing
>> tests.

I disagree here: disabling in Guix tests is _never_ done casually AFAIS
(as far as I see) but always ponderated and discussed, like in this case
;-)

[...]

> I think disabling specific tests is the least bad of these options.

Also: automated software testing is better than nothing but... who test
tests? *Sometime* it happens that tests introduces "collateral test
bugs" that have nothing to do with actual software issues, including
secutiry ones.

So IMHO neither upstream nor us should "blindly obey" to tests and
disable proved unreliable ones :-D

More on this specific issue in my next repy... :-)

[...]

Happy hacking! Gio'

-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Mark H Weaver  writes:

> Hi,
>
> Marius Bakke  writes:
>
>> Chris Marusich  writes:
>>
>>> Hi,
>>>
>>> I've been encountering this failure off and on for a few weeks now, and
>>> I'd like to help fix it.  In short, it seems like non-deterministic test
>>> failures, to me.  I think we should gather data and report the issue
>>> upstream, and maybe disable the offending tests in the meantime.
>>
>> I agree.  I notice many of these failing tests are for the TokuDB
>> backend, which I doubt anyone is using in Guix anyway.
>>
>> Here is a patch that disables all tests mentioned in this report.  I
>> would like to push it to core-updates.  Are there others?
>
> I'm concerned by how frequently and casually we simply disable failing
> tests.  What is the utility of running test suites at all, if this is
> how we respond?

I had no idea this issue was so widespread until I noticed Berlins
builders hit it more often than not.  I have not been able to reproduce
these failures on my machines.  So it was kind of a panic reaction,
being the person responsible for running these tests and all.

Looking further into the changes between 10.1.37 and 10.1.38, I notice
the 'tokudb.*' tests were enabled:

https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51

Watching the build on Berlin in real time, I also see that the test
output grind nearly to a halt while running those.
'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if
I'm reading the test output correctly.

The default test case timeout is 40 minutes (as specified in the Guix
package), but I'm using 80 for this build (60 was insufficient).

I suspect the problem is that the 'tokudb.*' tests put a lot of strain
on the file system, which causes these other tests to fail.  It's
interesting that disabling parallel build was insufficient though.


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Ludovic Courtès]

Hi Mark,

Mark H Weaver  skribis:

> Marius Bakke  writes:
>
>> Chris Marusich  writes:
>>
>>> Hi,
>>>
>>> I've been encountering this failure off and on for a few weeks now, and
>>> I'd like to help fix it.  In short, it seems like non-deterministic test
>>> failures, to me.  I think we should gather data and report the issue
>>> upstream, and maybe disable the offending tests in the meantime.
>>
>> I agree.  I notice many of these failing tests are for the TokuDB
>> backend, which I doubt anyone is using in Guix anyway.
>>
>> Here is a patch that disables all tests mentioned in this report.  I
>> would like to push it to core-updates.  Are there others?
>
> I'm concerned by how frequently and casually we simply disable failing
> tests.  What is the utility of running test suites at all, if this is
> how we respond?

I don’t think anyone is happy with that.  The alternative seems to be:
keeping an older version that perhaps didn’t have these problems but may
have known bugs and security issues, or keeping a package that fails to
build for a possibly long time.

I think disabling specific tests is the least bad of these options.  In
this case, we know that the offending tests relate to a specific
backend, and one can at least assume that potential issues are in that
area.  So I do think that this is an appropriate response.

Of course, in any such case, we should report the issue upstream, even
if we all too well know that non-deterministic test failures are hard to
address…

Ludo’.

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Hi,

Marius Bakke  writes:

> Chris Marusich  writes:
>
>> Hi,
>>
>> I've been encountering this failure off and on for a few weeks now, and
>> I'd like to help fix it.  In short, it seems like non-deterministic test
>> failures, to me.  I think we should gather data and report the issue
>> upstream, and maybe disable the offending tests in the meantime.
>
> I agree.  I notice many of these failing tests are for the TokuDB
> backend, which I doubt anyone is using in Guix anyway.
>
> Here is a patch that disables all tests mentioned in this report.  I
> would like to push it to core-updates.  Are there others?

I'm concerned by how frequently and casually we simply disable failing
tests.  What is the utility of running test suites at all, if this is
how we respond?

It makes me wonder how many programs are subtly broken on my Guix system
because of this widespread practice.

   Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Marius Bakke]

Chris Marusich  writes:

> Hi,
>
> I've been encountering this failure off and on for a few weeks now, and
> I'd like to help fix it.  In short, it seems like non-deterministic test
> failures, to me.  I think we should gather data and report the issue
> upstream, and maybe disable the offending tests in the meantime.

I agree.  I notice many of these failing tests are for the TokuDB
backend, which I doubt anyone is using in Guix anyway.

Here is a patch that disables all tests mentioned in this report.  I
would like to push it to core-updates.  Are there others?

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 578670e3c1..778c70eed0 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -704,8 +704,12 @@ Language.")
   ;; 2030-12-31.  See  for details.
   "main.mysqldump"
 
-  ;; XXX: Fails sporadically.
+  ;; XXX: These tests may fail on some hardware configurations,
+  ;; see  et al.
   "innodb_fts.crash_recovery"
+  "tokudb_alter_table.hcad_all_add"
+  "tokudb_bugs.mdev4533"
+  "tokudb_bugs.5733_innodb"
 
   ;; FIXME: This test fails on i686:
   ;; -myisampack: Can't create/write to file (Errcode: 17 "File exists")

WDYT?

Note that the latest MariaDB is 10.4.x, and these tests may well be
fixed in later versions.


signature.asc
Description: PGP signature

bug#35521: Mariadb test suite failures on x86_64-linux

[Chris Marusich]

Hi,

I've been encountering this failure off and on for a few weeks now, and
I'd like to help fix it.  In short, it seems like non-deterministic test
failures, to me.  I think we should gather data and report the issue
upstream, and maybe disable the offending tests in the meantime.

Mariadb failed for me earlier today with a different error than the ones
observed in this bug report so far.  My error was the following (when
building mariadb 10.1.40 on an x86_64-linux system using Guix 9b2644c):

  Failure: Failed 1/1990 tests, 99.95% were successful.

  Failing test(s): tokudb_bugs.5733_innodb

  The log files in var/log may give you some hint of what went wrong.

  If you want to report this error, please read first the documentation
  at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html

  558 tests were skipped, 169 by the test itself

I kept the failed build directory, but there is no "var" directory to be
found there.  I guess they meant system logs; I am not sure where such
logs would go when emitted from within a derivation.

The MySQL website suggested running mysql-test-run.pl with the --force
option, which I casually tried after invoking ". environment-variables"
from the failed build directory; however, it promptly failed because it
could not find 'my_safe_process' - maybe I didn't have everything set up
just so to run the tests manually.

Curiously, on a different x86_64-linux machine, using Guix commit
6c83c48 (which is only a few commits ahead of 9b2644c), I was able to
build mariadb successfully, although I am not sure when I built it
(running "guix build mariadb" currently results in quick success for me,
so on this machine I probably built or substituted it some time ago).
The derivation (without grafts) was identical to the one that failed to
build on the other machine, which is strange because I would normally
expect the same derivation to succeed on both machines.  For the record,
this was the derivation:

  $ guix build --no-grafts -d mariadb
  /gnu/store/9yw33r8r84qrsic7fiq0lqqkbzisv1cj-mariadb-10.1.40.drv

Perhaps these tests fail non-deterministically?  Or perhaps they fail in
a way that is specific something not isolated from the build process by
Guix, such as the kernel, the file system, or the hardware?

I tried to check the status of mariadb in Cuirass.  However, I only
found the following information:

  https://ci.guix.gnu.org/search?query=mariadb-10.1.40

For x86_64-linux, build 1304242 supposedly failed at 10 May 20:32 +0200
after about 3 hours of runtime:

  https://ci.guix.gnu.org/build/1304242/details

I say "supposedly failed" because I'm not sure why it failed.  The build
log seems to indicate no problems:

  https://ci.guix.gnu.org/build/1304242/log/raw

Has Cuirass tried to build mariadb since then?  May 10th was a long time
ago, and I am surprised there is not another build of it from master.

Mark H Weaver  writes:

> Mark H Weaver  writes:
>
>> The same build also failed twice in a row on my Thinkpad X200, and with
>> the same error each time, although it's a different error than happens
>> on hydra.gnunet.org.  On my X200, I get this instead:
>>
>>> Failure: Failed 1/1091 tests, 99.91% were successful.
>>> 
>>> Failing test(s): tokudb_bugs.mdev4533
>
> and it just failed a third time on my X200, again with the same error.

It seems like the tests may be flaky.  The test failure I saw was
different from yours.  And in my case, I actually was able to build (or
substitute) mariadb once.  So maybe what we need to do is gather enough
data to report the problem upstream, to enlist their help?

Platoxia  writes:

> This problem persists and is preventing sucessful completion of guix system 
> reconfigure for pre-1.0.0 systems (at least mine which is still at kernel 
> 4.20), not only for those using mariadb but also for anyone using any of the 
> 544 packages that depend on it; as per the command guix graph 
> --type=reverse-package mariadb | grep -c label).
>
> This could, potentially, be fixed by simply adding this test to the list of 
> disabled tests in the package definition:
>
> --- snip ---
> (add-after 'unpack 'adjust-tests
>(lambda _
>  (let ((disabled-tests
> '(;; These fail because root@hostname == root@localhost in
>   ;; the build environment, causing a user count mismatch.
>   ;; See .
>   "main.join_cache"
>   "main.explain_non_select"
>   "main.stat_tables_innodb"
>   "roles.acl_statistics"
>
>   ;; This file contains a time bomb which makes it fail 
> after
>   ;; 2030-12-31.  See  for 
> details.
>   "main.mysqldump"
>
>   ;; XXX: Fails sporadically.
>   "innodb_fts.crash_recovery"
>
>   ;; FIXME: 

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

Mark H Weaver  writes:

> The same build also failed twice in a row on my Thinkpad X200, and with
> the same error each time, although it's a different error than happens
> on hydra.gnunet.org.  On my X200, I get this instead:
>
>> Failure: Failed 1/1091 tests, 99.91% were successful.
>> 
>> Failing test(s): tokudb_bugs.mdev4533

and it just failed a third time on my X200, again with the same error.

  Mark

bug#35521: Mariadb test suite failures on x86_64-linux

[Mark H Weaver]

hydra.gnunet.org has failed to build mariadb on x86_64-linux
twice in a row:

  https://hydra.gnu.org/build/3475081#tabs-buildsteps

The same test failed both times:

> Failure: Failed 1/5075 tests, 99.98% were successful.
> 
> Failing test(s): tokudb_alter_table.hcad_all_add

The same build also failed twice in a row on my Thinkpad X200, and with
the same error each time, although it's a different error than happens
on hydra.gnunet.org.  On my X200, I get this instead:

> Failure: Failed 1/1091 tests, 99.91% were successful.
> 
> Failing test(s): tokudb_bugs.mdev4533

hydra.gnunet.org successfully built mariadb for i686-linux on its first
attempt:

  https://hydra.gnu.org/build/3473640

Here's the coresponding armhf-linux build, which has not yet been
attempted as I write this:

  https://hydra.gnu.org/build/3481309

   Mark