Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
> How about temporarily inserting something like the implementation below of
> Arnaud R's workaround somewhere (where?) in mariadb-server(-10.6).postinst?
>
> Thank you!
> Daniel Lewart
> Urbana, Illinois
> ---
> # Temporary workaround which should be removed after upstream fixes
> # https://jira.mariadb.org/browse/MDEV-28751
> fstype=$(findmnt -n -o FSTYPE -T /var/lib/mysql)
> if [ "$fstype" = overlay ]; then
> cat <<- EOF > "${mysql_cfgdir:-/etc/mysql}/mariadb.conf.d/90-live.cnf"
> [mysqld]
> innodb_flush_method = fsync
> EOF
> fi
Maybe just call it 90-overlayfs.cnf?
But I don't think this should be part of the maintainer scripts, maybe
something else? Feel free to open MR if you have a concrete
suggestion.
https://salsa.debian.org/mariadb-team/mariadb-server/-/wikis/Contributing-to-MariaDB-packaging-in-Debian
It would also be cool to have in the salsa-ci.yml a job that attempts
to use and run off overlayfs.
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Otto, et al,
How about temporarily inserting something like the implementation below of
Arnaud R's workaround somewhere (where?) in mariadb-server(-10.6).postinst?
Thank you!
Daniel Lewart
Urbana, Illinois
---
# Temporary workaround which should be removed after upstream fixes
# https://jira.mariadb.org/browse/MDEV-28751
fstype=$(findmnt -n -o FSTYPE -T /var/lib/mysql)
if [ "$fstype" = overlay ]; then
cat <<- EOF > "${mysql_cfgdir:-/etc/mysql}/mariadb.conf.d/90-live.cnf"
[mysqld]
innodb_flush_method = fsync
EOF
fi
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Otto, et al, > I don't see anything new in upstream > https://jira.mariadb.org/browse/MDEV-28751 about this. > However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can > for the sake of it check if the behaviour is still same on 10.11? Unfortunately, still the same. I created a live standard image with CODENAME=unstable: $ uname -a Linux debian 6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.7-1 (2023-01-18) x86_64 GNU/Linux Then I did the following: $ sudo apt -y install mariadb-server Output included: Setting up mariadb-server (1:10.11.1-1) ... Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service. Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145. Below is the output from the systemd journal. In the upstream bug: Daniel Black added a comment - 2022-06-07 04:50 I'll be just pushing this to the kernel (overlayfs) developers. I don't know what happened with that. The good news is that Arnaud R's upstream "innodb_flush_method = fsync" workaround still works. Sorry! Daniel Lewart Urbana, Illinois --- $ sudo journalctl -t mariadb-server.postinst | cut -c55- 2023-01-30 0:54:27 0 [Warning] InnoDB: Retry attempts for writing partial data failed. 2023-01-30 0:54:27 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at offset 0, 1048576 bytes should have been written, only 0 were written. Operating system error number 22. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded. 2023-01-30 0:54:27 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' 2023-01-30 0:54:27 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space 2023-01-30 0:54:27 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again. 2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 22 in a file operation. 2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' 2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation 230130 0:54:28 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.11.1-MariaDB-1 key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468018 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation Printing to addr2line failed /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ba61bd403e] /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x55ba61741c69] 2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90)[0x7f028b392f90] /lib/x86_64-linux-gnu/libc.so.6(+0x8accc)[0x7f028b3e1ccc] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f028b392ef2] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f028b37d472] 2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9 in a file operation. 2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor' 2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation /usr/sbin/mariadbd(+0x6692e1)[0x55ba613972e1] /usr/sbin/mariadbd(+0xd0a775)[0x55ba61a38775] /usr/sbin/mariadbd(+0xe140c8)[0x55ba61b420c8] /usr/sbin/mariadbd(+0xe14131)[0x55ba61b42131] /usr/sbin/mariadbd(+0xe16060)[0x55ba61b44060] /usr/sbin/mariadbd(+0xe17041)[0x55ba61b45041] /usr/sbin/mariadbd(+0xd7bbae)[0x55ba61aa9bae] /usr/sbin/mariadbd(+0xcb7545)[0x55ba619e5545]
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Hi! I don't see anything new in upstream https://jira.mariadb.org/browse/MDEV-28751 about this. However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can for the sake of it check if the behaviour is still same on 10.11? I would personally also be keen to figure out how to make the MariaDB datadir be able to run off overlayfs or something similar so I can implement my long time idea to have shadow upgrades where the old mariadbd and old data is kept running and intact, and every new version mariadbd would first run off overlayfs with real data to simulate the upgrade before it is done for real.
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Control: forwarded -1 https://jira.mariadb.org/browse/MDEV-28751 Hi Daniel! Thank you very much for your detailed report, your tests and to have filled a jira issue! I am now also subscribed to MDEV-28751 and will talk with Daniel Black to see how we can debug this and maybe check if this is an interesting step to add in our CI system. Cheers! -- Faustin signature.asc Description: PGP signature
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
tags + upstream Faustin, I found a much simpler way to demonstrate my problem with a Debian Bullseye Live image and MariaDB generic Linux server: * debian-live-11.3.0-amd64-standard.iso * mariadb-10.6.8-linux-systemd-x86_64.tar.gz I decided to file a new bug upstream: [DEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs https://jira.mariadb.org/browse/MDEV-28751 Also, strace shows a difference between 10.5.16 (good) and 10.6.8 (fails). Merci beaucoup, Dan Urbana, Illinois
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Faustin,
DL> I will give this a try. I'm curious what filesystem the container uses.
$ podman exec -it sys-test findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "fuse-overlayfs",
"fstype": "fuse.fuse-overlayfs",
"options":
"rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other"
}
]
}
Dan
Urbana, Illinois
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Faustin,
> I am not able to reproduce this with the latest
> mariadb-server-10.6-dbgsym version (10.6.8).
> Can you try with the latest version and tell me?
Yes, I tried 10.6.8-1 with the same result.
> Also, can you describe a bit better your setup? If this is a filesystem
> related issue, I would like to try to reproduce it myself.
Yes. This is a simplified version of my setup ...
1) sudo apt -y install apt-cacher-ng live-build
2) I used Live Build to create a live sid image:
#!/bin/sh
cd /dev/shm
sudo mount /dev/shm -odev,exec,remount,size=3G
lb config --apt-http-proxy http://localhost:3142 -d sid
echo '! Packages Priority standard' \
> config/package-lists/standard.list.chroot
echo 'deb http://deb.debian.org/debian-debug sid-debug main' \
> config/archives/live.list.binary
sudo lb build
3) qemu-system-x86_64 -enable-kvm -cdrom /dev/shm/live-image-amd64.hybrid.iso \
-cpu host -m 4G -smp 2
4) In the guest, it didn't matter what I tried:
sudo apt -y install mariadb-server
sudo apt -y install mariadb-server-10.6-dbgsym
mariadb-server-core-10.6-dbgsym
I'm sure it is an overlay filesystem issue:
$ sudo ls -l /var/lib/mysql/ibdata1
-rw-rw 1 mysql mysql 0 May 25 12:34 /var/lib/mysql/ibdata1
$ df -hT /var/lib/mysql
Filesystem Type Size Used Avail Use% Mounted on
overlayoverlay 2.0G 355M 1.6G 19% /
$ findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "overlay",
"fstype": "overlay",
"options":
"rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work"
}
]
}
> I have tested with a systemd container that I have created for these
> kind of testing, here are the commands (using podman):
>
> | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
> | podman exec -it sys-test bash -c "echo \"deb
> http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list
> && apt update && apt install -y mariadb-server-10.6-dbgsym"
I will give this a try. I'm curious what filesystem the container uses.
Thank you!
Dan
Urbana, IL
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
> > sudo rm -rf /var/lib/mysql/* > > sudo -u mysql mariadb-install-db Thanks Faustin, something was fixed between 10.6.7-3 and 10.6.8-1 that made mariadb-install-db succeed for me since today morning. With the March packages it kept failing every time during or after the InnoDB initialization (as I mentioned, I started clean every time, removing the mysql directories in /etc and /var).
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Hi Denys, I am not sure that the problem is the same, in your case it seems that the datadir does not exist or need to be recreated. If you are sure that nothing can be lost (or that you have backups), you should try the following: | sudo systemctl stop mariadb | sudo rm -rf /var/lib/mysql/* | sudo -u mysql mariadb-install-db | Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... | 2022-05-23 14:18:53 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1 | 2022-05-23 14:18:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF | 2022-05-23 14:18:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work. | OK | ... | sudo systemctl start mariadb Regards, Denys Nykula , 18/05/2022 – 17:16:00 (+0300): > Hello. The apt install of mariadb-server-10.6 fails for me too, > with the same line "Could not execute systemctl" but different > journal messages. I removed the existing setup in /etc and /var, > purged dependencies, but install still fails. How do I debug? > > systemd[1]: Starting MariaDB 10.6.7 database server... > [Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-3+b1) starting as process > 98614 ... > [Note] InnoDB: The first data file './ibdata1' did not exist. A new > tablespace will be created! > [Note] InnoDB: Compressed tables use zlib 1.2.11 > [Note] InnoDB: Number of pools: 1 > [Note] InnoDB: Using crc32 + pclmulqdq instructions > [Note] InnoDB: Using liburing > [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = > 134217728 > [Note] InnoDB: Completed initialization of buffer pool > [Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the > file full; Please wait ... > [Note] InnoDB: File './ibdata1' size is now 12 MB. > [Note] InnoDB: Setting log file ./ib_logfile101 size to 100663296 bytes > [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 > [Note] InnoDB: New log file created, LSN=10329 > [Note] InnoDB: Doublewrite buffer not found: creating new > [Note] InnoDB: Doublewrite buffer created > [Note] InnoDB: 128 rollback segments are active. > [Note] InnoDB: Creating shared tablespace for temporary tables > [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the > file full; Please wait ... > [Note] InnoDB: File './ibtmp1' size is now 12 MB. > [Note] InnoDB: 10.6.7 started; log sequence number 0; transaction id 3 > [Note] Plugin 'FEEDBACK' is disabled. > [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't > exist". Some plugins may be not loaded > [Warning] You need to use --log-bin to make --expire-logs-days or > --binlog-expire-logs-seconds work. > [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't > exist > [Note] Server socket created on IP: '127.0.0.1'. > [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' > doesn't exist > [ERROR] Aborting > Warning: Memory not freed: 280 > systemd[1]: mariadb.service: Main process exited, code=exited, > status=1/FAILURE > systemd[1]: mariadb.service: Failed with result 'exit-code'. > systemd[1]: Failed to start MariaDB 10.6.7 database server. > > ___ > pkg-mysql-maint mailing list > pkg-mysql-ma...@alioth-lists.debian.net > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint -- Faustin signature.asc Description: PGP signature
Bug#1009290: [debian-mysql] Bug#1009290: mariadb-server-10.6: Fails to start on live system
Hi Daniel! Thanks for your detailed report. I am not able to reproduce this with the latest mariadb-server-10.6-dbgsym version (10.6.8). Can you try with the latest version and tell me? Also, can you describe a bit better your setup? If this is a filesystem related issue, I would like to try to reproduce it myself. I have tested with a systemd container that I have created for these kind of testing, here are the commands (using podman): | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid | podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym" Cheers! Daniel Lewart , 11/04/2022 – 00:20:00 (-0500): > Package: mariadb-server-10.6 > Version: 1:10.6.7-3 > Severity: important > > Debian MariaDB Maintainers, > > Live Build (--distribution testing) image from April 2022. > 8 GB RAM, so should be plenty of disk space available. > > $ sudo apt install mariadb-server-10.6-dbgsym > ... > Setting up mariadb-server-10.6 (1:10.6.7-3) ... > Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → > /lib/systemd/system/mariadb.service. > Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. > ... > > $ sudo journalctl | grep -i mariadb | cut -c58- > [See below] > > Perhaps this is similar to: > #983002 - plocate: does not work with /var on overlayfs > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983002 > > Please let me know any additional information that you could use. > > Thank you! > Daniel Lewart > Urbana, Illinois > --- > 2022-04-10 22:53:02 0 [Warning] InnoDB: Retry attempts for writing partial > data failed. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at > offset 0, 1048576 bytes should have been written, only 0 were written. > Operating system error number 22. Check that your OS and file system support > files of this size. Check also that the disk is not full or a disk quota > exceeded. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Could not set the file size of > './ibdata1'. Probably out of disk space > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Database creation was aborted with > error Generic error. You may need to delete the ibdata1 file before trying to > start up again. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 22 in a > file operation. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument' > 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS > error 222. Cannot continue operation > 220410 22:53:02 [ERROR] mysqld got signal 6 ; > This could be because you hit a bug. It is also possible that this binary > or one of the libraries it was linked against is corrupt, improperly built, > or misconfigured. This error can also be caused by malfunctioning hardware. > > To report this bug, see https://mariadb.com/kb/en/reporting-bugs > > We will try our best to scrape up some info that will hopefully help > diagnose the problem, but since we have already crashed, > something is definitely wrong and this may fail. > > Server version: 10.6.7-MariaDB-3 > key_buffer_size=134217728 > read_buffer_size=131072 > max_used_connections=0 > max_threads=153 > thread_count=0 > It is possible that mysqld could use up to > key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467957 > K bytes of memory > Hope that's ok; if not, decrease some variables in the equation. > > Thread pointer: 0x0 > Attempting backtrace. You can use the following information to find out > where mysqld died. If you see no messages after this, something went > terribly wrong... > stack_bottom = 0x0 thread_stack 0x49000 > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a > file operation. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file > descriptor' > 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS > error 209. Cannot continue operation > Printing to addr2line failed > /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x555d0aa77f0e] > /usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x555d0a5c5268] > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a > file operation. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file > descriptor' > 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS > error 209. Cannot continue operation > /lib/x86_64-linux-gnu/libpthread.so.0(+0x12200)[0x7f606ba96200] > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a > file operation. > 2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file > descriptor' > 2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS > error 209. Cannot continue operation >
