I had an accident last night. I tried to delete a lot of rows from a production database in one transaction. I killed the transaction, and I didn't realise it was still rolling back an hour later when I tried to reboot the system for updates.
I might be wrong about exactly who is doing what, but I'll do my best to explain what happened, and then give a few notes about how I think the process could be improved. I don't specifically know if it's systemctl or the service file, or mysqld causing the issues. systemctl tried to shutdown mysqld, but due to the rollback, it wouldn't exit with anything less than kill -9. So, systemctl entered the failed state, however, in addition to that, it appears it closed the log output of the mysqld process so I couldn't see any of the output from the daemon. Eventually, after checking all our backups, I decided to issue kill -9 to mysqld. I then decided to try restarting the daemon using systemctl. It did start up, the log output showed the crash recovery procedure, but because it entered into the rollback recovery, systemctl never considered that the process had finished starting up, and then tried to kill it again, which failed (only kill -9 would work in this case). Again, log output was closed. I issued kill -9 again, and started up the process manually `sudo -u mysql mysqld` and let it go through the entire recovery process, and then rebooted the system. Normalcy restored. I think that the following things could be investigated/improved: - Log output shouldn't be closed if the daemon can't be killed. I'm not sure who is doing this - perhaps it's mysqld, or perhaps it's the service file, or perhaps its systemctl? - If a daemon fails to start up, trying to kill it.. may not be the best option. It's probably a matter of the systemctl service file detecting that a rollback is in progress and accepting that as a valid startup state, but I'm not really sure. In any case, I ended up having to do this process manually. Kind regards, Samuel _______________________________________________ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel