https://bugs.kde.org/show_bug.cgi?id=522845
Bug ID: 522845
Summary: Akonadi kills MariaDB too quickly — 3-second shutdown
timeout causes crashes on login/logout
Classification: Frameworks and Libraries
Product: Akonadi
Version First unspecified
Reported In:
Platform: Fedora RPMs
OS: Linux
Status: REPORTED
Severity: crash
Priority: NOR
Component: server
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected]
Target Milestone: ---
Hello,
I am the maintainer of MariaDB and MySQL packages in Fedora. I am receiving
crashes reports against MariaDB, but I traced them to you. Before reporting I
asked the MariaDB upstream developers to check whether the crashes aren't
caused by a bug in the MariaDB server, but it doesn't seem so.
I myself am not an expert in configuring MariaDB, but I can very much recommend
you to discuss with MariaDB upstream developers e.g. on their Zulip web forum
platform (mariadb.zulipchat.com), in case of any doubt.
DESCRIPTION
Multiple Fedora users report that `mariadbd` (launched by Akonadi) crashes with
SIGABRT during login or logout. MariaDB upstream developers analyzed the
multi-thread backtraces and confirmed that all crashes are caused by an
external signal killing the server, not by an internal MariaDB bug.
There are two likely contributing factors in Akonadi:
1. `stopInternalServer()` has a 3-second timeout before escalation
https://invent.kde.org/pim/akonadi/-/blob/master/src/server/storage/dbconfigmysql.cpp
| mDatabaseProcess->terminate();
| const bool result = mDatabaseProcess->waitForFinished(3000);
| if (!result) {
| mDatabaseProcess->kill();
| }
The `mCleanServerShutdownCommand` (`mysqladmin shutdown`) path also has only a
3-second `waitForFinished`. InnoDB needs time to flush dirty pages, finish
pending AIO operations, and destroy libaio contexts. 3 seconds is not enough
for a clean shutdown, especially under I/O load.
For comparison, MariaDB's own systemd service ships with `TimeoutStartSec=900`
and `TimeoutStopSec=900`, though I guess Akonadi usage won't need quite as
much.
2. `akonadi_control.service` has `TimeoutSec=5sec`
https://invent.kde.org/pim/akonadi/-/blob/master/src/akonadicontrol/akonadi_control.service.in
| TimeoutSec=5sec
This means systemd gives the entire `akonadi_control` process (and its cgroup,
including the MariaDB child) only 5 seconds to stop. Even if the code timeouts
were increased, systemd would kill everything after 5 seconds.
3. `mysql-global.conf` is missing `skip_slave_start`
https://invent.kde.org/pim/akonadi/-/blob/master/src/server/storage/mysql-global.conf
The Akonadi MariaDB configuration does not include `skip_slave_start`. This
causes MariaDB to initialize replication slave threads on startup, which is
unnecessary (Akonadi AFAIK does not use replication) and adds to startup time.
Two of the crash reports show the server killed while blocked inside
`init_slave()` → `start_slave_deadlock_handler_thread()`.
Adding `skip_slave_start` alongside the existing `skip_grant_tables` and
`skip_networking` would be consistent — all three disable server features that
are irrelevant for a local single-user Akonadi database.
OBSERVED CRASH PATTERNS
All crashes occur with `mariadbd` launched by Akonadi as a per-user instance
with `skip_grant_tables`, `skip_networking`, InnoDB default config.
Crash during startup — server killed while initializing (2 reporters, one says
"always reproducible"):
| handle_fatal_signal signal_handler.cc:310
| pthread_cond_wait
| start_slave_deadlock_handler_thread slave.cc:568
| init_slave slave.cc:672
| mysqld_main mysqld.cc:6186
Crash during shutdown — server killed while waiting for AIO writes to finish
(7+ reporters):
| handle_fatal_signal signal_handler.cc:310
| condition_variable::wait
| tpool::cache<tpool::aiocb>::wait tpool/tpool_structs.h:188
| io_slots::wait os/os0file.cc:111
| os_aio_wait_until_no_pending_writes_low os0file.cc:3187
Crash during shutdown — server killed during libaio context destruction (1
reporter):
| handle_fatal_signal signal_handler.cc:310
| io_destroy io_destroy.c:23
| aio_libaio::~aio_libaio tpool/aio_libaio.cc:170
| tpool::thread_pool::disable_aio tpool/tpool.h:286
| os_aio_free os/os0file.cc:3176
| innodb_shutdown srv/srv0start.cc:2103
UPSTREAM MariaDB ANALYSIS
MariaDB developers (Daniel Black, Vladislav Vaintroub) analyzed the backtraces
and concluded:
- "SIGABRT is from external — there's no abort functions in the stacks."
- "It looks like Akonadi is seriously rushing the shutdown of MariaDB."
- "Could it be that Akonadi gets impatient at slow start, and kills the server
with ABRT?"
- "skip_slave_start would be something logical to include in the configuration
file."
Discussion:
https://mariadb.zulipchat.com/#narrow/channel/118759-general/topic/.5BFedora.5D.20Akonadi.20.2B.20MariaDB.2011.2E8.20crash
MDEV-40293: https://jira.mariadb.org/browse/MDEV-40293
SUGGESTED FIXES
1. Increase `waitForFinished()` timeout in `stopInternalServer()` from 3000ms
to at least 30000ms
2. Increase `TimeoutSec` in `akonadi_control.service.in` from `5sec` to at
least `60sec` (or split into `TimeoutStartSec=10sec` / `TimeoutStopSec=60sec`)
3. Add `skip_slave_start` to `mysql-global.conf` (next to `skip_grant_tables`
and `skip_networking`)
AFFECTED VERSIONS
MariaDB 11.8.6, 11.8.8 on Fedora 44/45, but the issue is not version-specific —
any MariaDB version that takes more than 3 seconds to start or stop will be
affected.
Fedora Bugzilla tickets:
- https://bugzilla.redhat.com/show_bug.cgi?id=2461899 (AIO wait, 7+ reporters)
- https://bugzilla.redhat.com/show_bug.cgi?id=2487558 (AIO teardown, 1
reporter)
- https://bugzilla.redhat.com/show_bug.cgi?id=2496076 (init_slave, 1 reporter)
- https://bugzilla.redhat.com/show_bug.cgi?id=2496978 (init_slave, 1 reporter,
"always reproducible")
- https://bugzilla.redhat.com/show_bug.cgi?id=2476560 (bootstrap, 1 reporter,
rawhide)
--
You are receiving this mail because:
You are watching all bug changes.
