Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
Control: retitle -1 mariadb-plugin-gssapi-server: crash on partial upgrade of libk5crypto3 Filed now https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1036055 for krb5 maintainers to advise on.
Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
After some experimentation I found out that updating libkrb5-3 so that /usr/lib/x86_64-linux-gnu/libkrb5.so.3 upgrades will stop MariaDB from crashing. $ apt install libk5crypto3 libkrb5-3 ... libssl3 amd64 3.0.8-1 libgssapi-krb5-2 amd64 1.20.1-1+b1 libkrb5support0 amd64 1.20.1-1+b1 libkrb5-3 amd64 1.20.1-1+b1 I made a quick bash one-liner to install all packages from 'apt upgrade' step one-by-one to find out which one exactly causes MariaDB to crash, and the result was also libk5crypto3. Here is a more complete stack trace with debug symbols installed and addr2line working: 230514 5:19:28 [ERROR] mysqld got signal 11 ; Server version: 10.5.19-MariaDB-0+deb11u2 source revision: f8a85af8ca1c937b8d4f847477bd282f80251cde key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467880 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 2023-05-14 5:19:28 0 [Note] InnoDB: Buffer pool(s) load completed at 230514 5:19:28 ??:0(my_print_stacktrace)[0x556f175e616e] ??:0(handle_fatal_signal)[0x556f170def45] libc_sigaction.c:0(__restore_rt)[0x7f443ea8cf90] krb/prng.c:151(krb5_c_random_os_entropy)[0x7f443cd95de0] ??:0(krb5_init_context_profile)[0x7f443ce06cbf] ??:0(plugin_init())[0x7f443cee7569] /usr/lib/mysql/plugin/auth_gssapi.so(+0x2330)[0x7f443cee7330] ??:0(sys_var_pluginvar::sys_var_pluginvar(sys_var_chain*, char const*, st_plugin_int*, st_mysql_sys_var*, char const*))[0x556f16ef3562] ??:0(plugin_init(int*, char**, int))[0x556f16ef44d7] ??:0(unireg_abort)[0x556f16e1c0e6] ??:0(mysqld_main(int, char**))[0x556f16e21dc4] x86/libc-start.c:74(__libc_start_call_main)[0x7f443ea7818a] csu/libc-start.c:128(call_init)[0x7f443ea78245] ??:0(_start)[0x556f16e16daa] I think the solution would be for libk5crypto3 to have instead of Depends: libc6 (>= 2.33), libkrb5support0 (>= 1.16) to have Depends: libc6 (>= 2.33), libkrb5support0 (>= 1.20)
Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
Hi! I was able to reproduce this with: apt install mariadb-plugin-gssapi-server mariadb-plugin-gssapi-client sed s/bullseye/bookworm/g -i /etc/apt/sources.list apt update apt-get upgrade Upgrade only updates mysql-common and mariadb-common: $ dpkg -l | grep -e mysql -e maria ii libdbd-mariadb-perl1.21-3 ii libmariadb3:amd64 1:10.5.19-0+deb11u2 ii mariadb-client-10.51:10.5.19-0+deb11u2 ii mariadb-client-core-10.5 1:10.5.19-0+deb11u2 ii mariadb-common 1:10.11.2-1 ii mariadb-plugin-gssapi-client:amd64 1:10.5.19-0+deb11u2 ii mariadb-plugin-gssapi-server 1:10.5.19-0+deb11u2 iF mariadb-server-10.51:10.5.19-0+deb11u2 ii mariadb-server-core-10.5 1:10.5.19-0+deb11u2 ii mysql-common 5.8+1.1.0 I applied manually the debugging patch I have in https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/26 on /etc/init.d/mariadb and it resulted in: $ /etc/init.d/mariadb restart Stopping MariaDB database server: mariadbd. Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230514 01:36:00 mysqld_safe Can't log to error log and syslog at the same time. Remove all --log-error configuration options for --syslog to take effect. 230514 01:36:00 mysqld_safe Logging to '/tmp/tmp.AQ6kA9uzuH.err'. 230514 01:36:00 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql Running '/etc/init.d/mariadb start' failed with error log: 230514 01:36:00 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql 2023-05-14 1:36:00 0 [Note] Starting MariaDB 10.5.19-MariaDB-0+deb11u2 source revision f8a85af8ca1c937b8d4f847477bd282f80251cde as process 9189 2023-05-14 1:36:00 0 [Note] InnoDB: Uses event mutexes 2023-05-14 1:36:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2023-05-14 1:36:00 0 [Note] InnoDB: Number of pools: 1 2023-05-14 1:36:00 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2023-05-14 1:36:00 0 [Note] InnoDB: Using Linux native AIO 2023-05-14 1:36:00 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 2023-05-14 1:36:00 0 [Note] InnoDB: Completed initialization of buffer pool 2023-05-14 1:36:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=45127,45127 2023-05-14 1:36:00 0 [Note] InnoDB: 128 rollback segments are active. 2023-05-14 1:36:00 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2023-05-14 1:36:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2023-05-14 1:36:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2023-05-14 1:36:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2023-05-14 1:36:00 0 [Note] InnoDB: 10.5.19 started; log sequence number 45163; transaction id 20 2023-05-14 1:36:00 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2023-05-14 1:36:00 0 [Note] Plugin 'FEEDBACK' is disabled. 230514 1:36:00 [ERROR] mysqld got signal 11 ; ... /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5613bb07516e] /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5613bab6df45] /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90)[0x7f4b7a2faf90] /usr/lib/x86_64-linux-gnu/libk5crypto.so.3(krb5_c_random_os_entropy+0x0)[0x7f4b785fcde0] /usr/lib/x86_64-linux-gnu/libkrb5.so.3(krb5_init_context_profile+0x22f)[0x7f4b7866dcbf] /usr/lib/mysql/plugin/auth_gssapi.so(_Z11plugin_initv+0x129)[0x7f4b7874e569] /usr/lib/mysql/plugin/auth_gssapi.so(+0x2330)[0x7f4b7874e330] /usr/sbin/mariadbd(+0x77b562)[0x5613ba982562] /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x887)[0x5613ba9834d7] /usr/sbin/mariadbd(+0x6a40e6)[0x5613ba8ab0e6] /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x3f4)[0x5613ba8b0dc4] /lib/x86_64-linux-gnu/libc.so.6(+0x2718a)[0x7f4b7a2e618a] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f4b7a2e6245] /usr/sbin/mariadbd(_start+0x2a)[0x5613ba8a5daa] I suspect that some of the dependencies on the system (libc?) that updated causes GSSAPI to crash.
Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
On 11/05/2023 14.58, Otto Kekäläinen wrote: Thanks for reporting! Is this sporadic or does it reproduce every time? Just reran it with the same result, so seems deterministic. If you tell me what information to collect, I can easily enter the chroot and test things after the failure occurred. We have this upgrade scenario in CI without issues, thus asking about reproducibility. Do you use mariadb-plugin-gssapi-server? Andreas
Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
Thanks for reporting! Is this sporadic or does it reproduce every time? We have this upgrade scenario in CI without issues, thus asking about reproducibility.
Bug#1035944: mariadb: upgrade issue: mariadb-server-10.5 fails to restart after upgrading mariadb-common
Source: mariadb Version: 1:10.11.2-1 Severity: serious User: debian...@lists.debian.org Usertags: piuparts Control: affects -1 + mariadb-plugin-gssapi-server Hi, during a test with piuparts I noticed your package fails to upgrade from 'bullseye'. It installed fine in 'bullseye', then the upgrade to 'bookworm' fails. This test was planned to do a 2-stage upgrade: apt-get upgrade && apt-get distupgrade (the first part does not install any new packages, that will happen in the second part) but the first part already failed. I try to summarize what seems to happen: * In a bullseye chroot (which alles starting services) install mariadb-plugin-gssapi-server * switch sources.list to bookworm, run 'apt-get update' * run 'apt-get upgrade' - this only upgrades mysql-common and mariadb-common - this triggers a mariadb-server-10.5 restart which fails >From the attached log (scroll to the bottom...): ... Preparing to unpack .../24-mysql-common_5.8+1.1.0_all.deb ... Unpacking mysql-common (5.8+1.1.0) over (5.8+1.0.7) ... Preparing to unpack .../25-mariadb-common_1%3a10.11.2-1_all.deb ... Unpacking mariadb-common (1:10.11.2-1) over (1:10.5.19-0+deb11u2) ... ... Setting up mysql-common (5.8+1.1.0) ... ... Setting up mariadb-common (1:10.11.2-1) ... ... Processing triggers for mariadb-server-10.5 (1:10.5.19-0+deb11u2) ... invoke-rc.d: could not determine current runlevel Stopping MariaDB database server: mariadbd. Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [ESC][31mfailed![ESC][0m invoke-rc.d: initscript mariadb, action "restart" failed. dpkg: error processing package mariadb-server-10.5 (--configure): installed mariadb-server-10.5 package post-installation script subprocess returned error exit status 1 Processing triggers for debianutils (5.7-0.4) ... Processing triggers for libc-bin (2.36-9) ... Errors were encountered while processing: mariadb-server-10.5 This might indicate that mariadb-common needs a Breaks: mariadb-server-10.5 I cannot reproduce this by testing default-mariadb-server in the same way, so mariadb-plugin-gssapi-server seems to be needed to trigger that bug. cheers, Andreas mariadb-plugin-gssapi-server_1:10.11.2-1.log.gz Description: application/gzip
