Hi Kristian,
Kristian Nielsen schreef op 2026-07-08 10:54:
William David Edwards via discuss <[email protected]> writes:
What happened on 2026-04-02:
- 14:05 CET: application gets `SQLSTATE[42S02] ... Table
'app_db.users' doesn't exist` (error 1146). User confirms via
phpMyAdmin that the database `app_db` still exists but has no
tables.
- ~14:11 CET: everything works again. Data is intact (!)
On disk:
- All 12 .frm and all 12 .ibd files have Birth timestamps clustered in
a window of a few seconds, at `2026-04-02 14:11:30.xxx`.
- Order per table is .frm first, .ibd a few ms later. Example for one
table:
- foo.frm Birth `14:11:30.152`
- foo.ibd Birth `14:11:30.184`
- No #sql-* orphan files anywhere in the datadir.
Right, it's hard to know from the information available. But I would
say the
most likely explanation looks to be something happened to the
filesystem
from outside of the database server. That would be consistent with the
tables becoming invisible (database/table visibility is based on
filesystem
state IIRC), and with the file timestamps being so close. At least it's
a
place to look for other ideas: cron jobs, special filesystem / storage
system properties, etc.
Agreed that the filesystem would be a logical place to look, but I've
kind of excluded that because this database server hosts a few tens of
databases, and only `app_db` was problematic.
There are no cron jobs, systemd timers etc. that manually manipulate the
data dir. Here's all crons running on the system:
```
/etc/cron.d:
0 0 * * * root [ -d "/run/systemd/system" ] ||
/usr/share/atop/atop.daily&
30 3 * * 0 root test -e /run/systemd/system || SERVICE_MODE=1
/usr/libexec/e2fsprogs/e2scrub_all_cron
10 3 * * * root test -e /run/systemd/system || SERVICE_MODE=1
/sbin/e2scrub_all -A -r
09,39 * * * * root [ -x /usr/lib/php/sessionclean ] && if [ !
-d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi
13 5 * * 5 root /usr/bin/tuxis-cron
/etc/cronscripts/monitor-apt-keys.cron
51 4 * * * cluster-configuration-manager /usr/bin/tuxis-cron
/etc/cronscripts/purge-event-bus-publishes.cron
25 18 * * * root /usr/bin/tuxis-cron
/etc/cronscripts/update-ip-ranges-ferm-variables.cron
53 8 * * * queue-support /usr/bin/tuxis-cron
/etc/cronscripts/purge-queues.cron
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2
spool:
1 * * * * tuxis-cron update-databases-usage.cron
0 */24 * * * tuxis-cron mysql-backup.cron
0 0 * * 1 tuxis-cron mysql-optimize.cron
```
The only one running at minute 5 is sysstat, which has nothing to do
with MariaDB.
The only one that does *something* with databases is
`update-databases-usage.cron` (could have run at 14:01 and still at
14:05 - 14:11), which calls this code:
https://github.com/CyberfusionIO/python3-cyberfusion-database-support/blob/master/src/cyberfusion/DatabaseSupport/databases.py#L228
It's called for every database, every hour on over 100 servers, and does
not manipulate any database in any way.
The filesystem itself is about as simple as it can get: ext4. No LVM,
RAID and all that jazz.
We do do fstrim and freeze/thaw sometimes, and mariadb-dump, but not at
the involved timestamps.
Hope this helps,
- Kristian.
Met vriendelijke groeten,
William David Edwards
_______________________________________________
discuss mailing list -- [email protected]
To unsubscribe send an email to [email protected]
