Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 23 apr. 2013, om 23:03 heeft George McCollister george.mccollis...@gmail.com het volgende geschreven: On 04/20/2013 08:04 AM, Koen Kooi wrote: Op 18 apr. 2013, om 19:18 heeft George McCollistergeorge.mccollis...@gmail.com het volgende geschreven: On 04/10/2013 12:03 PM, Koen Kooi wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. I'm having exactly the same problem with sshd.socket (openssh) with systemd 197. I've done a netstat after it dies (just says dead no useful information) and port 22 still shows up under listening. systemctl start sshd.socket fixes the problem. I just upgraded to systemd 201 so I'll let you know if the problem shows up again. The problem happens intermittently so its been a bit elusive. It is indeed elusive, it hasn't happened to me in the past week, so progress is slow on this. ¸ regards, Koen This is really strange but I think I just accidentally found a way to reproduce the problem. 1) Reboot 2) Verify ssh works 3) login as root and run: systemctl --system daemon-reload Can't ssh anymore. If I do 'systemctl start sshd.socket' I can ssh again and doing 'systemctl --system daemon-reload' again doesn't seem to break it. I did a daemon reload the past week as well after reboot and yesterday my workstation suspended and dropped the connection, so I had to reconnect, which failed. regards, Koen ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On 04/20/2013 08:04 AM, Koen Kooi wrote: Op 18 apr. 2013, om 19:18 heeft George McCollistergeorge.mccollis...@gmail.com het volgende geschreven: On 04/10/2013 12:03 PM, Koen Kooi wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. I'm having exactly the same problem with sshd.socket (openssh) with systemd 197. I've done a netstat after it dies (just says dead no useful information) and port 22 still shows up under listening. systemctl start sshd.socket fixes the problem. I just upgraded to systemd 201 so I'll let you know if the problem shows up again. The problem happens intermittently so its been a bit elusive. It is indeed elusive, it hasn't happened to me in the past week, so progress is slow on this. regards, Koen This is really strange but I think I just accidentally found a way to reproduce the problem. 1) Reboot 2) Verify ssh works 3) login as root and run: systemctl --system daemon-reload Can't ssh anymore. If I do 'systemctl start sshd.socket' I can ssh again and doing 'systemctl --system daemon-reload' again doesn't seem to break it. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 18 apr. 2013, om 19:18 heeft George McCollister george.mccollis...@gmail.com het volgende geschreven: On 04/10/2013 12:03 PM, Koen Kooi wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. I'm having exactly the same problem with sshd.socket (openssh) with systemd 197. I've done a netstat after it dies (just says dead no useful information) and port 22 still shows up under listening. systemctl start sshd.socket fixes the problem. I just upgraded to systemd 201 so I'll let you know if the problem shows up again. The problem happens intermittently so its been a bit elusive. It is indeed elusive, it hasn't happened to me in the past week, so progress is slow on this. regards, Koen ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Am 18.04.2013 16:44, schrieb Lennart Poettering: On Thu, 18.04.13 16:38, Thomas Bächler (tho...@archlinux.org) wrote: Am 18.04.2013 16:04, schrieb Lennart Poettering: That said, screen should probably set up a new PAM session of its own and detach from the original one. That sounds like a good idea - unfortunately, screen does not seem to have PAM session support at all, and I couldn't find the obvious place to add this inside screen (I didn't invest much time in it either). This would allow screen sessions to be tracked by logind instead of having many State=closing sessions floating around. BTW, this probably needs sokme changes in systemd too. Since right now, if you open a PAM session from a process that is already part of a PAM session logind will just return the same data as the first time. So to make this work properly, we'd to define a way how certain PAM services can explicitly ask to get a new session assigned. So that's what happens with 'su' and 'sudo' - I always wondered. The obvious solution would be to add an option to pam_systemd which would force opening a new session despite being part of an active session. That setting could then be set in /etc/pam.d/screen. signature.asc Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 17 apr. 2013, om 21:05 heeft Lennart Poettering lenn...@poettering.net het volgende geschreven: On Wed, 10.04.13 19:03, Koen Kooi (k...@dominion.thruhere.net) wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. So, two things: When a service continuously dies we'll put the listening socket into fail state eventually. But you can see these ones easily in systemctl status, since they have a specific result service-failed-permanent. (results are shown next to the Active: field if a unit failed). And if the somebody invokes shutdown() on the listening socket (not the connection socket), but that's a really weird thing to do. But people do weird things, and this has occured before. Otherwise I have no idea what could have happened. Any chance you can reproduce this with strace attached to PID 1 or so? Still trying to reproduce it in a way I can instrument it. Is dropbear forked off one instance per connection, or one instance for all? Looks like one instance per connection. But I'm going to replace dropbear with openssh in the medium term because dropbear doesn't do enough PAM to register itself with logind, so things like 'screen' get killed on logout. regards, Koen ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Thu, 18.04.13 09:28, Koen Kooi (k...@dominion.thruhere.net) wrote: And if the somebody invokes shutdown() on the listening socket (not the connection socket), but that's a really weird thing to do. But people do weird things, and this has occured before. Otherwise I have no idea what could have happened. Any chance you can reproduce this with strace attached to PID 1 or so? Still trying to reproduce it in a way I can instrument it. Is dropbear forked off one instance per connection, or one instance for all? Looks like one instance per connection. Hmm, if so dropbear would never get access to the listening socket, to issue a shutdown() on it. This becomes weirder and weirder. If you manage to reproduce it, it would be good to get the full output of systemctl show -a for the socket unit. But I'm going to replace dropbear with openssh in the medium term because dropbear doesn't do enough PAM to register itself with logind, so things like 'screen' get killed on logout. You could work around this by use KillMode=process. That said, screen should probably set up a new PAM session of its own and detach from the original one. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Am 18.04.2013 16:04, schrieb Lennart Poettering: That said, screen should probably set up a new PAM session of its own and detach from the original one. That sounds like a good idea - unfortunately, screen does not seem to have PAM session support at all, and I couldn't find the obvious place to add this inside screen (I didn't invest much time in it either). This would allow screen sessions to be tracked by logind instead of having many State=closing sessions floating around. Has anyone worked on this? signature.asc Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Thu, 18.04.13 16:38, Thomas Bächler (tho...@archlinux.org) wrote: Am 18.04.2013 16:04, schrieb Lennart Poettering: That said, screen should probably set up a new PAM session of its own and detach from the original one. That sounds like a good idea - unfortunately, screen does not seem to have PAM session support at all, and I couldn't find the obvious place to add this inside screen (I didn't invest much time in it either). This would allow screen sessions to be tracked by logind instead of having many State=closing sessions floating around. BTW, this probably needs sokme changes in systemd too. Since right now, if you open a PAM session from a process that is already part of a PAM session logind will just return the same data as the first time. So to make this work properly, we'd to define a way how certain PAM services can explicitly ask to get a new session assigned. Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On 04/10/2013 12:03 PM, Koen Kooi wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. regards, Koen ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel I'm having exactly the same problem with sshd.socket (openssh) with systemd 197. I've done a netstat after it dies (just says dead no useful information) and port 22 still shows up under listening. systemctl start sshd.socket fixes the problem. I just upgraded to systemd 201 so I'll let you know if the problem shows up again. The problem happens intermittently so its been a bit elusive. Regards, George McCollister ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, 10.04.13 19:03, Koen Kooi (k...@dominion.thruhere.net) wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. So, two things: When a service continuously dies we'll put the listening socket into fail state eventually. But you can see these ones easily in systemctl status, since they have a specific result service-failed-permanent. (results are shown next to the Active: field if a unit failed). And if the somebody invokes shutdown() on the listening socket (not the connection socket), but that's a really weird thing to do. But people do weird things, and this has occured before. Otherwise I have no idea what could have happened. Any chance you can reproduce this with strace attached to PID 1 or so? Is dropbear forked off one instance per connection, or one instance for all? Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 11 apr. 2013, om 21:09 heeft David Strauss da...@davidstrauss.net het volgende geschreven: On Wed, Apr 10, 2013 at 11:00 PM, Koen Kooi k...@dominion.thruhere.net wrote: restarting it once it fails Is it the socket or the service? socket ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
2013/4/11 Koen Kooi k...@dominion.thruhere.net: On Wed, Apr 10, 2013 at 11:00 PM, Koen Kooi k...@dominion.thruhere.net wrote: restarting it once it fails Is it the socket or the service? socket I believe I experienced this with PHP-FPM when I injected the sockets by abusing PHP-FPM's internal env var method for preserving sockets during a configuration reload. We didn't not study the problem in detail, but I did raise it with Lennart, especially after removing PHP-FPM's socket cleanup routine fixed the issue. Lennart's pretty confident that an application closing a socket on shutdown shouldn't break systemd's ability to listen. Still, there must be something else going on. -- David Strauss | da...@davidstrauss.net | +1 512 577 5827 [mobile] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 10 apr. 2013, om 22:20 heeft Kok, Auke-jan H auke-jan.h@intel.com het volgende geschreven: On Wed, Apr 10, 2013 at 1:12 PM, David Strauss da...@davidstrauss.net wrote: Are you sure it's not the corresponding service that really failed? actually, that's a good point, but if the socket unit is dead, I assume that systemd no longer is bind()ed to the ports... Koen, can you verify that that is the case? I can try, I'll leave a board up a few days and see what happens. FWIW, restarting it once it fails is enough to fix it: root@soekris:/media/# systemctl status dropbear.socket dropbear.socket Loaded: loaded (/lib/systemd/system/dropbear.socket; enabled) Active: active (listening) since Tue 2013-03-26 20:43:06 CET; 2 weeks 1 days ago Accepted: 5; Connected: 2 CGroup: name=systemd:/system/dropbear.socket ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, Apr 10, 2013 at 11:00 PM, Koen Kooi k...@dominion.thruhere.net wrote: restarting it once it fails Is it the socket or the service? -- David Strauss | da...@davidstrauss.net | +1 512 577 5827 [mobile] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On 04/10/2013 05:03 PM, Koen Kooi wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, but till someone figures that out I'm looking to a way to automatically restart the socket when it fails, kinda like Restart=Always for services. Is such a thing possible? This is with 195 and 196, haven't tried 201 yet. When tackling things like this it's properly better to use timer unit that checks it periodically if the service responds ( some times they are up but in weird state ) and restarts it if it fails JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, Apr 10, 2013 at 8:03 PM, Koen Kooi k...@dominion.thruhere.net wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, After death, does `systemctl status dropbear.socket` show any error messages in the status line? -- Mantas Mikulėnas graw...@gmail.com ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Op 10 apr. 2013, om 19:13 heeft Mantas Mikulėnas graw...@gmail.com het volgende geschreven: On Wed, Apr 10, 2013 at 8:03 PM, Koen Kooi k...@dominion.thruhere.net wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, After death, does `systemctl status dropbear.socket` show any error messages in the status line? No, only that it's dead, so systemd knows that it's has failed somehow regards, Koen ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, Apr 10, 2013 at 11:10 AM, Koen Kooi k...@dominion.thruhere.net wrote: Op 10 apr. 2013, om 19:13 heeft Mantas Mikulėnas graw...@gmail.com het volgende geschreven: On Wed, Apr 10, 2013 at 8:03 PM, Koen Kooi k...@dominion.thruhere.net wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, After death, does `systemctl status dropbear.socket` show any error messages in the status line? No, only that it's dead, so systemd knows that it's has failed somehow dead isn't necessarily the same as failed... and I wonder if you can use ExecStartPost= here to kick it - it's valid for [Socket] sections. Auke ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
Are you sure it's not the corresponding service that really failed? On Wed, Apr 10, 2013 at 11:46 AM, Kok, Auke-jan H auke-jan.h@intel.com wrote: On Wed, Apr 10, 2013 at 11:10 AM, Koen Kooi k...@dominion.thruhere.net wrote: Op 10 apr. 2013, om 19:13 heeft Mantas Mikulėnas graw...@gmail.com het volgende geschreven: On Wed, Apr 10, 2013 at 8:03 PM, Koen Kooi k...@dominion.thruhere.net wrote: Hi, I have a bit of a heisenbug where dropbear.socket will just die and needs a systemctl restart dropbear.socket. I can't tell why it's dying, just that it does within 3 days of uptime. After restarting it it seems to be rock solid again for at least some weeks. The real way to fix this is to find out why it dies, After death, does `systemctl status dropbear.socket` show any error messages in the status line? No, only that it's dead, so systemd knows that it's has failed somehow dead isn't necessarily the same as failed... and I wonder if you can use ExecStartPost= here to kick it - it's valid for [Socket] sections. Auke ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- David Strauss | da...@davidstrauss.net | +1 512 577 5827 [mobile] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, Apr 10, 2013 at 1:12 PM, David Strauss da...@davidstrauss.net wrote: Are you sure it's not the corresponding service that really failed? actually, that's a good point, but if the socket unit is dead, I assume that systemd no longer is bind()ed to the ports... Koen, can you verify that that is the case? Auke ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Socket is dying, how to automatically restart it?
On Wed, Apr 10, 2013 at 1:20 PM, Kok, Auke-jan H auke-jan.h@intel.com wrote: actually, that's a good point, but if the socket unit is dead, I assume that systemd no longer is bind()ed to the ports... I'd have to look at source or run some experiments for details, but I know systemd's socket listeners can get into a funk when the corresponding service is marked as failed. -- David Strauss | da...@davidstrauss.net | +1 512 577 5827 [mobile] ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
