Re: [systemd-devel] Shutdown problems
On Mon, 24.11.14 12:31, Nikolaus Rath (nikol...@rath.org) wrote: Sorry for the late reply, still have a huge backlog of mail which I am trying to process right now. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. Hmm, this is weird. If there are no shutdown hooks and no initrd that installs /run/initramfs/shutdown, then the only thing between your hook and the reboot() syscall is an invocation of sync(). I wonder if that might be the issue here? Can you try adding an explicit /usr/bin/sync invocation to your hook script, and check if that survives? (add an echo line after it, to check if it completes). In general though: this really smells like an LVM or kernel issue. I usually just reassign bugs involving LVM to the LVM package. I'd recommend asking the LVM people for help, maybe? Any chance you can try to reproduce this without LVM in the mix? Sorry if I have no other recommendations, but if sync() and reboot() are really the only things that are executed after the hook script then I really have no suggestion what might be going wrong except that there's something wrong with the kernel... Sorry if this isn't too helpful, Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Nikolaus Rath nikol...@rath.org writes: On 11/13/2014 12:54 PM, Nikolaus Rath wrote: Nikolaus Rath nikol...@rath.org writes: Lennart Poettering lenn...@poettering.net writes: On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Is there some indication that LVN is at fault? As I said in my first email, the crucial difference seems to be if an X11 console is active or not: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. Some more information: * if I start a debug-shell on a serial port, at some point the shell seems to freeze as well. * if I boot with sysvinit instead of systemd things work fine: - The system reboots even if an X11 console is active - During the shutdown, I can switch between text consoles and see log messages Any ideas? On a whim, I also tried blacklisting the nouveau module and using the integrated graphics with the i915 module instead. It did not make a difference. *ping*. Really no one able to help with this? Thanks, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Nikolaus Rath wrote on 16/11/14 16:54: No one able to help at all? Please be patient. You've only left it a couple days since your last reply and only one of those days was a weekday. People will likely reply soon enough. I would however recommend you disable things like plymouth and remove the splash and quiet keywords from the kernel command line to see more debugging. Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Nikolaus Rath nikol...@rath.org writes: On 11/13/2014 12:54 PM, Nikolaus Rath wrote: Nikolaus Rath nikol...@rath.org writes: Lennart Poettering lenn...@poettering.net writes: On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Is there some indication that LVN is at fault? As I said in my first email, the crucial difference seems to be if an X11 console is active or not: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. Some more information: * if I start a debug-shell on a serial port, at some point the shell seems to freeze as well. * if I boot with sysvinit instead of systemd things work fine: - The system reboots even if an X11 console is active - During the shutdown, I can switch between text consoles and see log messages Any ideas? On a whim, I also tried blacklisting the nouveau module and using the integrated graphics with the i915 module instead. It did not make a difference. No one able to help at all? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Nikolaus Rath nikol...@rath.org writes: Lennart Poettering lenn...@poettering.net writes: On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Is there some indication that LVN is at fault? As I said in my first email, the crucial difference seems to be if an X11 console is active or not: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. Some more information: * if I start a debug-shell on a serial port, at some point the shell seems to freeze as well. * if I boot with sysvinit instead of systemd things work fine: - The system reboots even if an X11 console is active - During the shutdown, I can switch between text consoles and see log messages Any ideas? Thanks, Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
On 11/13/2014 12:54 PM, Nikolaus Rath wrote: Nikolaus Rath nikol...@rath.org writes: Lennart Poettering lenn...@poettering.net writes: On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Is there some indication that LVN is at fault? As I said in my first email, the crucial difference seems to be if an X11 console is active or not: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. Some more information: * if I start a debug-shell on a serial port, at some point the shell seems to freeze as well. * if I boot with sysvinit instead of systemd things work fine: - The system reboots even if an X11 console is active - During the shutdown, I can switch between text consoles and see log messages Any ideas? On a whim, I also tried blacklisting the nouveau module and using the integrated graphics with the i915 module instead. It did not make a difference. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Lennart Poettering lenn...@poettering.net writes: On Sat, 08.11.14 11:16, Nikolaus Rath (nikol...@rath.org) wrote: Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Hmm the logs show that systemd pretty much completed its shutdown. After the message Cannot finalize remaining DM devices, continuing. the only thing that still runs is the shutdown hooks you used to generate this log, plus either a jump back into your initrd (if your initrd supports that) or the reboot() system call. If the latter hangs then it's a kernel bug. reboot -f works fine - could it still be a kernel bug? Please check if there are any other scripts in /lib/systemd/system-shutdown/ that might be at fault here. Nope, none. Please check if your initrd is one of those which support jumping back into the initrd on shutdown. For that check if /run/initramfs/shutdown exists during runtime and is executable. No, /run/initramfs/shutdown does not exist. If so, it's probably an initrd problem, please file a bug against your initrd implementation. You appear to be using LVM, I wouldn't be surprised if LVM is broken here, but I cannot help you debugging this, please contact the LVM maintainers in this case. Is there some indication that LVN is at fault? As I said in my first email, the crucial difference seems to be if an X11 console is active or not: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Lennart Poettering lenn...@poettering.net writes: On Thu, 06.11.14 19:07, Nikolaus Rath (nikol...@rath.org) wrote: Hello, I'm having some trouble shutting down my system with systemd. What happens is the following: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. I tried the debugging technique from http://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1 and created a file debug.sh in /lib/systemd/system-shutdown with permisson 755 and the following contents: $ cat /lib/systemd/system-shutdown/debug.sh #!/bin/sh exec /shutdown.log exec 21 mount -o remount,rw / echo stdout echo stderr 2 dmesg systemctl list-jobs systemctl status echo done mount -o remount,ro / You open the file for writing before you actually mount the fs read-only. That cannot work. However, no file /shutdown.log is created after reboot. I also tried placing debug.sh in /usr/lib/systemd/system-shutdown instead (this is a Debian system), but this did not work either. I also tried enabling the debug-shell, but this did not help because I can't access tty9 when the hang occurs. I also tried running an sshd process in the debug shell (to see if I can still reach the system remotely when it hangs), but the connection got closed when I ran the systemctl command - why might that be? I had hoped that processes spawned from the debug-shell would not get killed? I am using systemd 215-5+b1 on Debian jessie. I also have plymouth enabled. Anyone able to help? Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Here's the output (obtained by changing log level and remounting earlier in the debug.sh script): https://dl.dropboxusercontent.com/u/11545826/shutdown.log Thanks for your help! Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
Am 2014-11-07 04:07, schrieb Nikolaus Rath: $ cat /lib/systemd/system-shutdown/debug.sh #!/bin/sh exec /shutdown.log exec 21 mount -o remount,rw / Well, you need to mount / rewrite *before* redirecting output into a file. Try putting the 'mount -o remount,rw /' line to the top of the script and try again. Christian ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Shutdown problems
On Thu, 06.11.14 19:07, Nikolaus Rath (nikol...@rath.org) wrote: Hello, I'm having some trouble shutting down my system with systemd. What happens is the following: * If I execute systemctl reboot while a text console is active, everything works fine. * If I execute systemctl reboot while the X11 console is active, the system hangs (I tried waiting up to 7 minutes). Furthermore, I am unable to switch to another console with Ctrl+Alt+Fn, the computer becomes unresponsive to the keyboard and the monitor powers down. On which tty/pty systemctl itself is executed does not matter (I tested this by running systemctl in an ssh session from a remote system), it only matters which console is currently active. I tried the debugging technique from http://freedesktop.org/wiki/Software/systemd/Debugging/#index2h1 and created a file debug.sh in /lib/systemd/system-shutdown with permisson 755 and the following contents: $ cat /lib/systemd/system-shutdown/debug.sh #!/bin/sh exec /shutdown.log exec 21 mount -o remount,rw / echo stdout echo stderr 2 dmesg systemctl list-jobs systemctl status echo done mount -o remount,ro / You open the file for writing before you actually mount the fs read-only. That cannot work. However, no file /shutdown.log is created after reboot. I also tried placing debug.sh in /usr/lib/systemd/system-shutdown instead (this is a Debian system), but this did not work either. I also tried enabling the debug-shell, but this did not help because I can't access tty9 when the hang occurs. I also tried running an sshd process in the debug shell (to see if I can still reach the system remotely when it hangs), but the connection got closed when I ran the systemctl command - why might that be? I had hoped that processes spawned from the debug-shell would not get killed? I am using systemd 215-5+b1 on Debian jessie. I also have plymouth enabled. Anyone able to help? Please boot with systemd.log_level=debug, then make the machine hang and check what the last things in the logs say. Maybe then paste that somewhere online and post the URL for that here, so that we can have a look. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
