Package: mariadb-server-core
Version: 1:10.11.11-0+deb12u1
Severity: critical
Tags: upstream
Justification: causes serious data loss
X-Debbugs-Cc: [email protected]
Dear Maintainer,
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation?
Upgraded to mariadb (1:10.11.11-0+deb12u1) bookworm;
after that so did these crashes start to happen.
Previews version of mariadb in debian bookworm worked fine.
* What exactly did you do (or not do) that was effective (or
ineffective)?
No workaround so far.
This Mariadb crash happens on multiple servers with different hardware.
Uppstream repport:
https://jira.mariadb.org/projects/MDEV/issues/MDEV-35424?filter=allopenissues
The maraidb settings for the servers in this report.
thread_stack = 192K
thread_cache_size = 8
max_connections = 600
query_cache_size = 0
query_cache_type = 0
max_heap_table_size = 128M
tmp_table_size = 128M
read_buffer_size = 512K
innodb_buffer_pool_size = 12G
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 4M
innodb_flush_method = O_DIRECT
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
join_buffer_size = 2M
table_definition_cache = 10000
* What was the outcome of this action?
Crash of mariadb on first web9 server.
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61628 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61462 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61447 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61602 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61351 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61342 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:40:12 web9 mariadbd[3888776]: 2025-03-24 7:40:12 0 [ERROR] [FATAL]
InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 24 07:40:12 web9 mariadbd[3888776]: 250324 7:40:12 [ERROR]
/usr/sbin/mariadbd got signal 6 ;
Mar 24 07:40:12 web9 mariadbd[3888776]: Sorry, we probably made a mistake, and
this is a bug.
Mar 24 07:40:12 web9 mariadbd[3888776]: Your assistance in bug reporting will
enable us to fix this for the next release.
Mar 24 07:40:12 web9 mariadbd[3888776]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 24 07:40:12 web9 mariadbd[3888776]: a bug on https://jira.mariadb.org/.
Mar 24 07:40:12 web9 mariadbd[3888776]: Please include the information from the
server start above, to the end of the
Mar 24 07:40:12 web9 mariadbd[3888776]: information below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Server version:
10.11.11-MariaDB-0+deb12u1 source revision:
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 24 07:40:12 web9 mariadbd[3888776]: The information page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 24 07:40:12 web9 mariadbd[3888776]: contains instructions to obtain a
better version of the backtrace below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Following these instructions will help
MariaDB developers provide a fix quicker.
Mar 24 07:40:12 web9 mariadbd[3888776]: Attempting backtrace. Include this in
the bug report.
Mar 24 07:40:12 web9 mariadbd[3888776]: (note: Retrieving this information may
fail)
Mar 24 07:40:12 web9 mariadbd[3888776]: Thread pointer: 0x0
Mar 24 07:40:12 web9 mariadbd[3888776]: stack_bottom = 0x0 thread_stack 0x30000
Mar 24 07:40:13 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5576c94642ee]
Mar 24 07:40:13 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x5576c8f42e29]
Mar 24 07:40:14 web9 mariadbd[3888776]: 2025-03-24 7:40:14 0 [Note] InnoDB:
Memory pressure event freed 771915 pages
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f52b305b050]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x8aeec)[0x7f52b30a9eec]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f52b305afb2]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f52b3045472]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(+0x6c323b)[0x5576c8b5f23b]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(+0x6bb7fa)[0x5576c8b577fa]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x5576c93fa378]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x5576c93fac6f]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x4f)[0x5576c93f967f]
Mar 24 07:40:16 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f52b32d44a3]
Mar 24 07:40:17 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f52b30a81f5]
Mar 24 07:40:17 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x10989c)[0x7f52b312889c]
Mar 24 07:40:17 web9 mariadbd[3888776]: Writing a core file...
Mar 24 07:40:17 web9 mariadbd[3888776]: Working directory at /var/lib/mysql
Mar 24 07:40:17 web9 mariadbd[3888776]: Resource Limits (excludes unlimited
resources):
Mar 24 07:40:17 web9 mariadbd[3888776]: Limit Soft Limit
Hard Limit Units
Mar 24 07:40:17 web9 mariadbd[3888776]: Max stack size 8388608
unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max core file size 0
unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max processes 256997
256997 processes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max open files 32768
32768 files
Mar 24 07:40:17 web9 mariadbd[3888776]: Max locked memory 524288
524288 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max pending signals 256997
256997 signals
Mar 24 07:40:17 web9 mariadbd[3888776]: Max msgqueue size 819200
819200 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max nice priority 0
0
Mar 24 07:40:17 web9 mariadbd[3888776]: Max realtime priority 0
0
Mar 24 07:40:17 web9 mariadbd[3888776]: Core pattern: core
Mar 24 07:40:17 web9 mariadbd[3888776]: Kernel version: Linux version
6.1.0-32-amd64 ([email protected]) (gcc-12 (Debian 12.2.0-14)
12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.129-1 (2025-03-06)
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Main process exited,
code=killed, status=6/ABRT
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU
time.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Scheduled restart job,
restart counter is at 1.
Mar 24 07:40:24 web9 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11
database server.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU
time.
Crash of mariadb on second web8 server.
Mar 22 10:55:13 web8 mariadbd[3951656]: 2025-03-22 10:55:13 0 [ERROR] [FATAL]
InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:13 web8 mariadbd[3951656]: 250322 10:55:13 [ERROR]
/usr/sbin/mariadbd got signal 6 ;
Mar 22 10:55:13 web8 mariadbd[3951656]: Sorry, we probably made a mistake, and
this is a bug.
Mar 22 10:55:13 web8 mariadbd[3951656]: Your assistance in bug reporting will
enable us to fix this for the next release.
Mar 22 10:55:13 web8 mariadbd[3951656]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 22 10:55:13 web8 mariadbd[3951656]: a bug on https://jira.mariadb.org/.
Mar 22 10:55:13 web8 mariadbd[3951656]: Please include the information from the
server start above, to the end of the
Mar 22 10:55:13 web8 mariadbd[3951656]: information below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Server version:
10.11.11-MariaDB-0+deb12u1 source revision:
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 22 10:55:13 web8 mariadbd[3951656]: The information page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 22 10:55:13 web8 mariadbd[3951656]: contains instructions to obtain a
better version of the backtrace below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Following these instructions will help
MariaDB developers provide a fix quicker.
Mar 22 10:55:13 web8 mariadbd[3951656]: Attempting backtrace. Include this in
the bug report.
Mar 22 10:55:13 web8 mariadbd[3951656]: (note: Retrieving this information may
fail)
Mar 22 10:55:13 web8 mariadbd[3951656]: Thread pointer: 0x0
Mar 22 10:55:13 web8 mariadbd[3951656]: stack_bottom = 0x0 thread_stack 0x30000
Mar 22 10:55:16 web8 mariadbd[3951656]: 2025-03-22 10:55:16 0 [Note] InnoDB:
Memory pressure event freed 387350 pages
Mar 22 10:55:21 web8 mariadbd[3951656]:
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ed836892ee]
Mar 22 10:55:21 web8 mariadbd[3951656]:
/usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55ed83167e29]
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 0 [Note] InnoDB:
Memory pressure event freed 90 pages
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 418105 [ERROR]
[FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for
dict_sys.latch. Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Main process exited,
code=killed, status=6/ABRT
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s
CPU time.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Scheduled restart job,
restart counter is at 1.
Mar 22 10:55:32 web8 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11
database server.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s
CPU time.
Mar 22 10:55:32 web8 systemd[1]: Starting mariadb.service - MariaDB 10.11.11
database server...
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] Starting
MariaDB 10.11.11-MariaDB-0+deb12u1 source revision
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid
Xmmv8P9Fsl7fwijTPqiP1aMaxa8= as process 200075
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Compressed tables use zlib 1.2.13
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Number of transaction pools: 1
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Using crc32 + pclmulqdq instructions
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Using liburing
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Initializing buffer pool, total size = 6.000GiB, chunk size = 96.000MiB
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Initialized memory pressure event listener
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Completed initialization of buffer pool
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
File system buffers for log disabled (block size=512 bytes)
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Starting crash recovery from checkpoint LSN=2376864813498
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: End
of log at LSN=2376872752547
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: To
recover: 1923 pages
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: 128
rollback segments are active.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Removed temporary tablespace data file: "./ibtmp1"
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full;
Please wait ...
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
File './ibtmp1' size is now 12.000MiB.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: log
sequence number 2376872752547; transaction id 7237064055
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Plugin
'FEEDBACK' is disabled.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Warning] You need
to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds
work.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Server
socket created on IP: '127.0.0.1'.
Mar 22 10:55:34 web8 mariadbd[200075]: 2025-03-22 10:55:34 0 [Note]
/usr/sbin/mariadbd: ready for connections.
Mar 22 10:55:34 web8 mariadbd[200075]: Version: '10.11.11-MariaDB-0+deb12u1'
socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 12
Mar 22 10:55:34 web8 systemd[1]: Started mariadb.service - MariaDB 10.11.11
database server.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200167]: Upgrading MySQL tables if
necessary.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200170]: /usr/bin/mysql_upgrade:
the '--basedir' option is always ignored
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200190]: Checking for insecure
root accounts.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200196]: Triggering myisam-recover
for all MyISAM tables and aria-recover for all Aria tables
Mar 22 10:55:38 web8 mariadbd[200075]: 2025-03-22 10:55:38 0 [Note] InnoDB:
Memory pressure event freed 62433 pages
Mar 22 10:55:39 web8 mariadbd[200075]: 2025-03-22 10:55:39 0 [Note] InnoDB:
Buffer pool(s) load completed at 250322 10:55:39
* What outcome did you expect instead?
That Mariadb did not crash.
*** End of the template - remove these template lines ***
-- System Information:
Debian Release: 12.10
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-32-amd64 (SMP w/64 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server-core depends on:
ii libc6 2.36-9+deb12u10
ii libcrypt1 1:4.4.33-2
ii libgcc-s1 12.2.0-14
ii libnuma1 2.0.16-1
ii libpcre2-8-0 10.42-1
ii libssl3 3.0.15-1~deb12u1
ii libstdc++6 12.2.0-14
ii libsystemd0 252.36-1~deb12u1
ii liburing2 2.3-3
ii mariadb-common 1:10.11.11-0+deb12u1
ii zlib1g 1:1.2.13.dfsg-1
mariadb-server-core recommends no packages.
mariadb-server-core suggests no packages.
-- no debconf information
